2023-03-08 19:00:12,028 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4 2023-03-08 19:00:12,041 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple timeout: 13 mins 2023-03-08 19:00:12,057 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-08 19:00:12,058 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/cluster_ea2ca4d9-6781-beee-9c1a-51bfaa940573, deleteOnExit=true 2023-03-08 19:00:12,058 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-08 19:00:12,059 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/test.cache.data in system properties and HBase conf 2023-03-08 19:00:12,059 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/hadoop.tmp.dir in system properties and HBase conf 2023-03-08 19:00:12,060 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/hadoop.log.dir in system properties and HBase conf 2023-03-08 19:00:12,061 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-08 19:00:12,061 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-08 19:00:12,062 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-08 19:00:12,184 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-08 19:00:12,560 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-08 19:00:12,564 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-08 19:00:12,564 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-08 19:00:12,564 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-08 19:00:12,565 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-08 19:00:12,565 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-08 19:00:12,565 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-08 19:00:12,566 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-08 19:00:12,566 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-08 19:00:12,566 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-08 19:00:12,566 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/nfs.dump.dir in system properties and HBase conf 2023-03-08 19:00:12,567 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/java.io.tmpdir in system properties and HBase conf 2023-03-08 19:00:12,567 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-08 19:00:12,567 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-08 19:00:12,568 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-08 19:00:13,026 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-08 19:00:13,030 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-08 19:00:13,313 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-08 19:00:13,494 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-08 19:00:13,515 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-08 19:00:13,554 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-08 19:00:13,586 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/java.io.tmpdir/Jetty_localhost_45753_hdfs____pebvz2/webapp 2023-03-08 19:00:13,758 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45753 2023-03-08 19:00:13,770 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-08 19:00:13,770 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-08 19:00:14,186 WARN [Listener at localhost/34283] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-08 19:00:14,314 WARN [Listener at localhost/34283] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-08 19:00:14,338 WARN [Listener at localhost/34283] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-08 19:00:14,346 INFO [Listener at localhost/34283] log.Slf4jLog(67): jetty-6.1.26 2023-03-08 19:00:14,352 INFO [Listener at localhost/34283] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/java.io.tmpdir/Jetty_localhost_44957_datanode____.e2siny/webapp 2023-03-08 19:00:14,466 INFO [Listener at localhost/34283] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44957 2023-03-08 19:00:14,796 WARN [Listener at localhost/43773] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-08 19:00:14,809 WARN [Listener at localhost/43773] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-08 19:00:14,814 WARN [Listener at localhost/43773] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-08 19:00:14,817 INFO [Listener at localhost/43773] log.Slf4jLog(67): jetty-6.1.26 2023-03-08 19:00:14,823 INFO [Listener at localhost/43773] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/java.io.tmpdir/Jetty_localhost_41379_datanode____.mpean5/webapp 2023-03-08 19:00:14,955 INFO [Listener at localhost/43773] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41379 2023-03-08 19:00:14,975 WARN [Listener at localhost/43619] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-08 19:00:14,995 WARN [Listener at localhost/43619] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-08 19:00:14,999 WARN [Listener at localhost/43619] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-08 19:00:15,001 INFO [Listener at localhost/43619] log.Slf4jLog(67): jetty-6.1.26 2023-03-08 19:00:15,012 INFO [Listener at localhost/43619] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/java.io.tmpdir/Jetty_localhost_33201_datanode____afuri2/webapp 2023-03-08 19:00:15,149 INFO [Listener at localhost/43619] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33201 2023-03-08 19:00:15,170 WARN [Listener at localhost/45021] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-08 19:00:15,376 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x74c3c99ea4de8d1d: Processing first storage report for DS-6d50f5f3-dee3-4200-a565-87ad0d2ea3ec from datanode 24830eb0-72be-48b6-9e8c-ace0fc6eb977 2023-03-08 19:00:15,378 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x74c3c99ea4de8d1d: from storage DS-6d50f5f3-dee3-4200-a565-87ad0d2ea3ec node DatanodeRegistration(127.0.0.1:42379, datanodeUuid=24830eb0-72be-48b6-9e8c-ace0fc6eb977, infoPort=35633, infoSecurePort=0, ipcPort=43619, storageInfo=lv=-57;cid=testClusterID;nsid=954609056;c=1678302013103), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-03-08 19:00:15,379 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x971ea7c3de01426d: Processing first storage report for DS-142238fa-90dc-4cd1-9438-7f1639f03bd7 from datanode 9513f804-42c9-41da-863a-7ca390f59556 2023-03-08 19:00:15,379 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x971ea7c3de01426d: from storage DS-142238fa-90dc-4cd1-9438-7f1639f03bd7 node DatanodeRegistration(127.0.0.1:42929, datanodeUuid=9513f804-42c9-41da-863a-7ca390f59556, infoPort=34999, infoSecurePort=0, ipcPort=43773, storageInfo=lv=-57;cid=testClusterID;nsid=954609056;c=1678302013103), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-08 19:00:15,379 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x14393c076e57f683: Processing first storage report for DS-9ddae0ae-a29a-46d0-ad70-eae1c0ede482 from datanode 7368293d-a0de-467d-aff9-0b753c131e12 2023-03-08 19:00:15,379 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x14393c076e57f683: from storage DS-9ddae0ae-a29a-46d0-ad70-eae1c0ede482 node DatanodeRegistration(127.0.0.1:34881, datanodeUuid=7368293d-a0de-467d-aff9-0b753c131e12, infoPort=34369, infoSecurePort=0, ipcPort=45021, storageInfo=lv=-57;cid=testClusterID;nsid=954609056;c=1678302013103), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-08 19:00:15,379 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x74c3c99ea4de8d1d: Processing first storage report for DS-b6da74af-bb5c-4b1b-9dc7-312b87c15255 from datanode 24830eb0-72be-48b6-9e8c-ace0fc6eb977 2023-03-08 19:00:15,380 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x74c3c99ea4de8d1d: from storage DS-b6da74af-bb5c-4b1b-9dc7-312b87c15255 node DatanodeRegistration(127.0.0.1:42379, datanodeUuid=24830eb0-72be-48b6-9e8c-ace0fc6eb977, infoPort=35633, infoSecurePort=0, ipcPort=43619, storageInfo=lv=-57;cid=testClusterID;nsid=954609056;c=1678302013103), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-08 19:00:15,380 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x971ea7c3de01426d: Processing first storage report for DS-fe75043e-7770-4810-9dfe-9f927c5623d2 from datanode 9513f804-42c9-41da-863a-7ca390f59556 2023-03-08 19:00:15,380 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x971ea7c3de01426d: from storage DS-fe75043e-7770-4810-9dfe-9f927c5623d2 node DatanodeRegistration(127.0.0.1:42929, datanodeUuid=9513f804-42c9-41da-863a-7ca390f59556, infoPort=34999, infoSecurePort=0, ipcPort=43773, storageInfo=lv=-57;cid=testClusterID;nsid=954609056;c=1678302013103), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-08 19:00:15,380 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x14393c076e57f683: Processing first storage report for DS-563485e9-85de-4193-9e81-07c7285ff3d2 from datanode 7368293d-a0de-467d-aff9-0b753c131e12 2023-03-08 19:00:15,380 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x14393c076e57f683: from storage DS-563485e9-85de-4193-9e81-07c7285ff3d2 node DatanodeRegistration(127.0.0.1:34881, datanodeUuid=7368293d-a0de-467d-aff9-0b753c131e12, infoPort=34369, infoSecurePort=0, ipcPort=45021, storageInfo=lv=-57;cid=testClusterID;nsid=954609056;c=1678302013103), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-03-08 19:00:15,604 DEBUG [Listener at localhost/45021] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4 2023-03-08 19:00:15,683 INFO [Listener at localhost/45021] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/cluster_ea2ca4d9-6781-beee-9c1a-51bfaa940573/zookeeper_0, clientPort=59169, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/cluster_ea2ca4d9-6781-beee-9c1a-51bfaa940573/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/cluster_ea2ca4d9-6781-beee-9c1a-51bfaa940573/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-08 19:00:15,699 INFO [Listener at localhost/45021] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=59169 2023-03-08 19:00:15,710 INFO [Listener at localhost/45021] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-08 19:00:15,713 INFO [Listener at localhost/45021] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-08 19:00:16,422 INFO [Listener at localhost/45021] util.FSUtils(471): Created version file at hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1 with version=8 2023-03-08 19:00:16,422 INFO [Listener at localhost/45021] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/hbase-staging 2023-03-08 19:00:16,771 INFO [Listener at localhost/45021] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-08 19:00:17,304 INFO [Listener at localhost/45021] client.ConnectionUtils(127): master/jenkins-hbase6:0 server-side Connection retries=18 2023-03-08 19:00:17,338 INFO [Listener at localhost/45021] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-08 19:00:17,338 INFO [Listener at localhost/45021] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-08 19:00:17,339 INFO [Listener at localhost/45021] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-08 19:00:17,339 INFO [Listener at localhost/45021] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-08 19:00:17,339 INFO [Listener at localhost/45021] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-08 19:00:17,485 INFO [Listener at localhost/45021] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-08 19:00:17,581 DEBUG [Listener at localhost/45021] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-08 19:00:17,700 INFO [Listener at localhost/45021] ipc.NettyRpcServer(120): Bind to /172.31.2.20:43879 2023-03-08 19:00:17,710 INFO [Listener at localhost/45021] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-08 19:00:17,713 INFO [Listener at localhost/45021] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-08 19:00:17,735 INFO [Listener at localhost/45021] zookeeper.RecoverableZooKeeper(93): Process identifier=master:43879 connecting to ZooKeeper ensemble=127.0.0.1:59169 2023-03-08 19:00:17,781 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:438790x0, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-08 19:00:17,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:43879-0x101d3eda4f90000 connected 2023-03-08 19:00:17,827 DEBUG [Listener at localhost/45021] zookeeper.ZKUtil(164): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-08 19:00:17,828 DEBUG [Listener at localhost/45021] zookeeper.ZKUtil(164): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-08 19:00:17,832 DEBUG [Listener at localhost/45021] zookeeper.ZKUtil(164): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-08 19:00:17,842 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43879 2023-03-08 19:00:17,842 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43879 2023-03-08 19:00:17,843 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43879 2023-03-08 19:00:17,846 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43879 2023-03-08 19:00:17,847 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43879 2023-03-08 19:00:17,853 INFO [Listener at localhost/45021] master.HMaster(444): hbase.rootdir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1, hbase.cluster.distributed=false 2023-03-08 19:00:17,940 INFO [Listener at localhost/45021] client.ConnectionUtils(127): regionserver/jenkins-hbase6:0 server-side Connection retries=18 2023-03-08 19:00:17,940 INFO [Listener at localhost/45021] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-08 19:00:17,940 INFO [Listener at localhost/45021] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-08 19:00:17,940 INFO [Listener at localhost/45021] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-08 19:00:17,941 INFO [Listener at localhost/45021] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-08 19:00:17,941 INFO [Listener at localhost/45021] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-08 19:00:17,945 INFO [Listener at localhost/45021] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-08 19:00:17,949 INFO [Listener at localhost/45021] ipc.NettyRpcServer(120): Bind to /172.31.2.20:33875 2023-03-08 19:00:17,951 INFO [Listener at localhost/45021] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-08 19:00:17,958 INFO [Listener at localhost/45021] mob.MobFileCache(124): MobFileCache disabled 2023-03-08 19:00:17,959 INFO [Listener at localhost/45021] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-08 19:00:17,961 INFO [Listener at localhost/45021] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-08 19:00:17,963 INFO [Listener at localhost/45021] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33875 connecting to ZooKeeper ensemble=127.0.0.1:59169 2023-03-08 19:00:17,968 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:338750x0, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-08 19:00:17,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33875-0x101d3eda4f90001 connected 2023-03-08 19:00:17,969 DEBUG [Listener at localhost/45021] zookeeper.ZKUtil(164): regionserver:33875-0x101d3eda4f90001, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-08 19:00:17,974 DEBUG [Listener at localhost/45021] zookeeper.ZKUtil(164): regionserver:33875-0x101d3eda4f90001, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-08 19:00:17,978 DEBUG [Listener at localhost/45021] zookeeper.ZKUtil(164): regionserver:33875-0x101d3eda4f90001, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-08 19:00:17,980 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33875 2023-03-08 19:00:17,981 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33875 2023-03-08 19:00:17,982 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33875 2023-03-08 19:00:17,992 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33875 2023-03-08 19:00:17,992 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33875 2023-03-08 19:00:18,012 INFO [Listener at localhost/45021] client.ConnectionUtils(127): regionserver/jenkins-hbase6:0 server-side Connection retries=18 2023-03-08 19:00:18,012 INFO [Listener at localhost/45021] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-08 19:00:18,013 INFO [Listener at localhost/45021] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-08 19:00:18,013 INFO [Listener at localhost/45021] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-08 19:00:18,013 INFO [Listener at localhost/45021] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-08 19:00:18,014 INFO [Listener at localhost/45021] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-08 19:00:18,014 INFO [Listener at localhost/45021] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-08 19:00:18,017 INFO [Listener at localhost/45021] ipc.NettyRpcServer(120): Bind to /172.31.2.20:41161 2023-03-08 19:00:18,018 INFO [Listener at localhost/45021] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-08 19:00:18,024 INFO [Listener at localhost/45021] mob.MobFileCache(124): MobFileCache disabled 2023-03-08 19:00:18,025 INFO [Listener at localhost/45021] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-08 19:00:18,028 INFO [Listener at localhost/45021] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-08 19:00:18,030 INFO [Listener at localhost/45021] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41161 connecting to ZooKeeper ensemble=127.0.0.1:59169 2023-03-08 19:00:18,037 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:411610x0, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-08 19:00:18,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:41161-0x101d3eda4f90002 connected 2023-03-08 19:00:18,039 DEBUG [Listener at localhost/45021] zookeeper.ZKUtil(164): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-08 19:00:18,040 DEBUG [Listener at localhost/45021] zookeeper.ZKUtil(164): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-08 19:00:18,040 DEBUG [Listener at localhost/45021] zookeeper.ZKUtil(164): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-08 19:00:18,046 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41161 2023-03-08 19:00:18,047 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41161 2023-03-08 19:00:18,048 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41161 2023-03-08 19:00:18,050 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41161 2023-03-08 19:00:18,051 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41161 2023-03-08 19:00:18,070 INFO [Listener at localhost/45021] client.ConnectionUtils(127): regionserver/jenkins-hbase6:0 server-side Connection retries=18 2023-03-08 19:00:18,070 INFO [Listener at localhost/45021] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-08 19:00:18,070 INFO [Listener at localhost/45021] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-08 19:00:18,070 INFO [Listener at localhost/45021] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-08 19:00:18,070 INFO [Listener at localhost/45021] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-08 19:00:18,070 INFO [Listener at localhost/45021] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-08 19:00:18,071 INFO [Listener at localhost/45021] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-08 19:00:18,073 INFO [Listener at localhost/45021] ipc.NettyRpcServer(120): Bind to /172.31.2.20:46129 2023-03-08 19:00:18,073 INFO [Listener at localhost/45021] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-08 19:00:18,074 INFO [Listener at localhost/45021] mob.MobFileCache(124): MobFileCache disabled 2023-03-08 19:00:18,076 INFO [Listener at localhost/45021] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-08 19:00:18,078 INFO [Listener at localhost/45021] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-08 19:00:18,079 INFO [Listener at localhost/45021] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46129 connecting to ZooKeeper ensemble=127.0.0.1:59169 2023-03-08 19:00:18,085 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:461290x0, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-08 19:00:18,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:46129-0x101d3eda4f90003 connected 2023-03-08 19:00:18,086 DEBUG [Listener at localhost/45021] zookeeper.ZKUtil(164): regionserver:46129-0x101d3eda4f90003, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-08 19:00:18,087 DEBUG [Listener at localhost/45021] zookeeper.ZKUtil(164): regionserver:46129-0x101d3eda4f90003, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-08 19:00:18,088 DEBUG [Listener at localhost/45021] zookeeper.ZKUtil(164): regionserver:46129-0x101d3eda4f90003, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-08 19:00:18,088 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46129 2023-03-08 19:00:18,088 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46129 2023-03-08 19:00:18,089 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46129 2023-03-08 19:00:18,090 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46129 2023-03-08 19:00:18,090 DEBUG [Listener at localhost/45021] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46129 2023-03-08 19:00:18,092 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase6.apache.org,43879,1678302016597 2023-03-08 19:00:18,103 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-08 19:00:18,104 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase6.apache.org,43879,1678302016597 2023-03-08 19:00:18,125 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-08 19:00:18,125 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:33875-0x101d3eda4f90001, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-08 19:00:18,126 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:46129-0x101d3eda4f90003, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-08 19:00:18,125 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-08 19:00:18,126 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-08 19:00:18,130 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-08 19:00:18,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-08 19:00:18,132 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase6.apache.org,43879,1678302016597 from backup master directory 2023-03-08 19:00:18,135 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase6.apache.org,43879,1678302016597 2023-03-08 19:00:18,135 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-08 19:00:18,136 WARN [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-08 19:00:18,136 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase6.apache.org,43879,1678302016597 2023-03-08 19:00:18,139 INFO [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-08 19:00:18,145 INFO [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-08 19:00:18,276 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/hbase.id with ID: 43f3b849-ce2d-44b3-bf4e-d27a2bd1b3f0 2023-03-08 19:00:18,331 INFO [master/jenkins-hbase6:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-08 19:00:18,345 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-08 19:00:18,393 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x48334728 to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:18,424 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54a8b031, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:18,447 INFO [master/jenkins-hbase6:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-08 19:00:18,448 INFO [master/jenkins-hbase6:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-08 19:00:18,465 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-08 19:00:18,465 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-08 19:00:18,467 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-08 19:00:18,472 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-08 19:00:18,474 INFO [master/jenkins-hbase6:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-08 19:00:18,506 INFO [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/MasterData/data/master/store-tmp 2023-03-08 19:00:18,562 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:18,562 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-08 19:00:18,562 INFO [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-08 19:00:18,562 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-08 19:00:18,562 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-08 19:00:18,562 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-08 19:00:18,562 INFO [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-08 19:00:18,562 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-08 19:00:18,564 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/MasterData/WALs/jenkins-hbase6.apache.org,43879,1678302016597 2023-03-08 19:00:18,583 INFO [master/jenkins-hbase6:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase6.apache.org%2C43879%2C1678302016597, suffix=, logDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/MasterData/WALs/jenkins-hbase6.apache.org,43879,1678302016597, archiveDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/MasterData/oldWALs, maxLogs=10 2023-03-08 19:00:18,645 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42929,DS-142238fa-90dc-4cd1-9438-7f1639f03bd7,DISK] 2023-03-08 19:00:18,645 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34881,DS-9ddae0ae-a29a-46d0-ad70-eae1c0ede482,DISK] 2023-03-08 19:00:18,645 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42379,DS-6d50f5f3-dee3-4200-a565-87ad0d2ea3ec,DISK] 2023-03-08 19:00:18,656 DEBUG [RS-EventLoopGroup-5-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-08 19:00:18,738 INFO [master/jenkins-hbase6:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/MasterData/WALs/jenkins-hbase6.apache.org,43879,1678302016597/jenkins-hbase6.apache.org%2C43879%2C1678302016597.1678302018593 2023-03-08 19:00:18,738 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34881,DS-9ddae0ae-a29a-46d0-ad70-eae1c0ede482,DISK], DatanodeInfoWithStorage[127.0.0.1:42379,DS-6d50f5f3-dee3-4200-a565-87ad0d2ea3ec,DISK], DatanodeInfoWithStorage[127.0.0.1:42929,DS-142238fa-90dc-4cd1-9438-7f1639f03bd7,DISK]] 2023-03-08 19:00:18,740 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-08 19:00:18,741 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:18,745 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-08 19:00:18,746 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-08 19:00:18,827 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-08 19:00:18,835 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-08 19:00:18,859 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-08 19:00:18,870 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:18,876 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-08 19:00:18,877 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-08 19:00:18,892 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-08 19:00:18,896 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-08 19:00:18,897 INFO [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64907867, jitterRate=-0.03279741108417511}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:18,897 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-08 19:00:18,898 INFO [master/jenkins-hbase6:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-08 19:00:18,920 INFO [master/jenkins-hbase6:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-08 19:00:18,920 INFO [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-08 19:00:18,923 INFO [master/jenkins-hbase6:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-08 19:00:18,924 INFO [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-08 19:00:18,958 INFO [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 33 msec 2023-03-08 19:00:18,958 INFO [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-08 19:00:18,982 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-08 19:00:18,987 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-08 19:00:19,017 INFO [master/jenkins-hbase6:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-08 19:00:19,020 INFO [master/jenkins-hbase6:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-08 19:00:19,022 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-08 19:00:19,027 INFO [master/jenkins-hbase6:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-08 19:00:19,030 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-08 19:00:19,033 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-08 19:00:19,034 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-08 19:00:19,035 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-08 19:00:19,046 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-08 19:00:19,050 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:33875-0x101d3eda4f90001, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-08 19:00:19,050 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:46129-0x101d3eda4f90003, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-08 19:00:19,050 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-08 19:00:19,050 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-08 19:00:19,050 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-08 19:00:19,051 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase6.apache.org,43879,1678302016597, sessionid=0x101d3eda4f90000, setting cluster-up flag (Was=false) 2023-03-08 19:00:19,066 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-08 19:00:19,070 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-08 19:00:19,071 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase6.apache.org,43879,1678302016597 2023-03-08 19:00:19,078 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-08 19:00:19,083 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-08 19:00:19,084 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase6.apache.org,43879,1678302016597 2023-03-08 19:00:19,086 WARN [master/jenkins-hbase6:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp 2023-03-08 19:00:19,093 INFO [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(952): ClusterId : 43f3b849-ce2d-44b3-bf4e-d27a2bd1b3f0 2023-03-08 19:00:19,094 INFO [RS:2;jenkins-hbase6:46129] regionserver.HRegionServer(952): ClusterId : 43f3b849-ce2d-44b3-bf4e-d27a2bd1b3f0 2023-03-08 19:00:19,094 INFO [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(952): ClusterId : 43f3b849-ce2d-44b3-bf4e-d27a2bd1b3f0 2023-03-08 19:00:19,098 DEBUG [RS:2;jenkins-hbase6:46129] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-08 19:00:19,098 DEBUG [RS:1;jenkins-hbase6:41161] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-08 19:00:19,098 DEBUG [RS:0;jenkins-hbase6:33875] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-08 19:00:19,104 DEBUG [RS:2;jenkins-hbase6:46129] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-08 19:00:19,104 DEBUG [RS:1;jenkins-hbase6:41161] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-08 19:00:19,104 DEBUG [RS:0;jenkins-hbase6:33875] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-08 19:00:19,105 DEBUG [RS:1;jenkins-hbase6:41161] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-08 19:00:19,105 DEBUG [RS:2;jenkins-hbase6:46129] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-08 19:00:19,105 DEBUG [RS:0;jenkins-hbase6:33875] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-08 19:00:19,110 DEBUG [RS:1;jenkins-hbase6:41161] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-08 19:00:19,111 DEBUG [RS:1;jenkins-hbase6:41161] zookeeper.ReadOnlyZKClient(139): Connect 0x79ce09c8 to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:19,112 DEBUG [RS:2;jenkins-hbase6:46129] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-08 19:00:19,121 DEBUG [RS:0;jenkins-hbase6:33875] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-08 19:00:19,122 DEBUG [RS:2;jenkins-hbase6:46129] zookeeper.ReadOnlyZKClient(139): Connect 0x37d22f28 to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:19,123 DEBUG [RS:0;jenkins-hbase6:33875] zookeeper.ReadOnlyZKClient(139): Connect 0x3c108782 to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:19,127 DEBUG [RS:1;jenkins-hbase6:41161] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7709e881, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:19,128 DEBUG [RS:1;jenkins-hbase6:41161] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@733662b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase6.apache.org/172.31.2.20:0 2023-03-08 19:00:19,130 DEBUG [RS:2;jenkins-hbase6:46129] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3915d7a2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:19,130 DEBUG [RS:2;jenkins-hbase6:46129] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d2a4458, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase6.apache.org/172.31.2.20:0 2023-03-08 19:00:19,131 DEBUG [RS:0;jenkins-hbase6:33875] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32d77e70, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:19,132 DEBUG [RS:0;jenkins-hbase6:33875] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36e8ecf2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase6.apache.org/172.31.2.20:0 2023-03-08 19:00:19,151 DEBUG [RS:2;jenkins-hbase6:46129] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase6:46129 2023-03-08 19:00:19,152 DEBUG [RS:1;jenkins-hbase6:41161] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase6:41161 2023-03-08 19:00:19,155 INFO [RS:2;jenkins-hbase6:46129] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-08 19:00:19,155 INFO [RS:1;jenkins-hbase6:41161] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-08 19:00:19,155 INFO [RS:2;jenkins-hbase6:46129] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-08 19:00:19,155 INFO [RS:1;jenkins-hbase6:41161] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-08 19:00:19,156 DEBUG [RS:2;jenkins-hbase6:46129] regionserver.HRegionServer(1023): About to register with Master. 2023-03-08 19:00:19,156 DEBUG [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(1023): About to register with Master. 2023-03-08 19:00:19,157 DEBUG [RS:0;jenkins-hbase6:33875] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase6:33875 2023-03-08 19:00:19,157 INFO [RS:0;jenkins-hbase6:33875] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-08 19:00:19,157 INFO [RS:0;jenkins-hbase6:33875] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-08 19:00:19,157 DEBUG [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(1023): About to register with Master. 2023-03-08 19:00:19,158 INFO [RS:2;jenkins-hbase6:46129] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase6.apache.org,43879,1678302016597 with isa=jenkins-hbase6.apache.org/172.31.2.20:46129, startcode=1678302018069 2023-03-08 19:00:19,158 INFO [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase6.apache.org,43879,1678302016597 with isa=jenkins-hbase6.apache.org/172.31.2.20:41161, startcode=1678302018011 2023-03-08 19:00:19,158 INFO [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase6.apache.org,43879,1678302016597 with isa=jenkins-hbase6.apache.org/172.31.2.20:33875, startcode=1678302017939 2023-03-08 19:00:19,176 DEBUG [RS:2;jenkins-hbase6:46129] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-08 19:00:19,176 DEBUG [RS:1;jenkins-hbase6:41161] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-08 19:00:19,176 DEBUG [RS:0;jenkins-hbase6:33875] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-08 19:00:19,226 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-08 19:00:19,229 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:44431, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-03-08 19:00:19,229 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:38063, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-08 19:00:19,229 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:39465, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-03-08 19:00:19,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43879] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:19,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43879] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:19,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:19,239 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase6:0, corePoolSize=5, maxPoolSize=5 2023-03-08 19:00:19,240 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase6:0, corePoolSize=5, maxPoolSize=5 2023-03-08 19:00:19,240 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase6:0, corePoolSize=5, maxPoolSize=5 2023-03-08 19:00:19,240 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase6:0, corePoolSize=5, maxPoolSize=5 2023-03-08 19:00:19,240 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase6:0, corePoolSize=10, maxPoolSize=10 2023-03-08 19:00:19,240 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,240 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase6:0, corePoolSize=2, maxPoolSize=2 2023-03-08 19:00:19,240 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,241 INFO [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1678302049241 2023-03-08 19:00:19,243 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-08 19:00:19,249 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-08 19:00:19,249 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-08 19:00:19,254 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-08 19:00:19,255 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-08 19:00:19,263 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-08 19:00:19,264 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-08 19:00:19,264 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-08 19:00:19,264 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-08 19:00:19,265 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,267 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-08 19:00:19,268 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-08 19:00:19,269 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-08 19:00:19,270 DEBUG [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(2829): Master is not running yet 2023-03-08 19:00:19,270 DEBUG [RS:2;jenkins-hbase6:46129] regionserver.HRegionServer(2829): Master is not running yet 2023-03-08 19:00:19,270 WARN [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-03-08 19:00:19,270 DEBUG [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(2829): Master is not running yet 2023-03-08 19:00:19,270 WARN [RS:2;jenkins-hbase6:46129] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-03-08 19:00:19,271 WARN [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-03-08 19:00:19,272 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-08 19:00:19,272 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-08 19:00:19,273 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase6:0:becomeActiveMaster-HFileCleaner.large.0-1678302019273,5,FailOnTimeoutGroup] 2023-03-08 19:00:19,274 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase6:0:becomeActiveMaster-HFileCleaner.small.0-1678302019274,5,FailOnTimeoutGroup] 2023-03-08 19:00:19,274 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,274 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-08 19:00:19,275 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,275 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,297 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-08 19:00:19,298 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-08 19:00:19,299 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1 2023-03-08 19:00:19,320 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:19,322 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-08 19:00:19,324 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/info 2023-03-08 19:00:19,325 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-08 19:00:19,325 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:19,326 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-08 19:00:19,328 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/rep_barrier 2023-03-08 19:00:19,328 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-08 19:00:19,329 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:19,329 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-08 19:00:19,331 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/table 2023-03-08 19:00:19,332 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-08 19:00:19,333 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:19,334 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740 2023-03-08 19:00:19,335 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740 2023-03-08 19:00:19,339 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-03-08 19:00:19,341 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-08 19:00:19,345 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-08 19:00:19,346 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65895514, jitterRate=-0.018080323934555054}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-03-08 19:00:19,346 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-08 19:00:19,346 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-08 19:00:19,346 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-08 19:00:19,346 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-08 19:00:19,346 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-08 19:00:19,346 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-08 19:00:19,348 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-08 19:00:19,348 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-08 19:00:19,353 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-08 19:00:19,353 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-08 19:00:19,362 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-08 19:00:19,372 INFO [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase6.apache.org,43879,1678302016597 with isa=jenkins-hbase6.apache.org/172.31.2.20:33875, startcode=1678302017939 2023-03-08 19:00:19,372 INFO [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase6.apache.org,43879,1678302016597 with isa=jenkins-hbase6.apache.org/172.31.2.20:41161, startcode=1678302018011 2023-03-08 19:00:19,372 INFO [RS:2;jenkins-hbase6:46129] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase6.apache.org,43879,1678302016597 with isa=jenkins-hbase6.apache.org/172.31.2.20:46129, startcode=1678302018069 2023-03-08 19:00:19,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43879] master.ServerManager(394): Registering regionserver=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:19,379 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-08 19:00:19,379 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.ServerManager(394): Registering regionserver=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:19,380 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43879] master.ServerManager(394): Registering regionserver=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:19,386 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:19,388 DEBUG [RS:2;jenkins-hbase6:46129] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1 2023-03-08 19:00:19,388 DEBUG [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1 2023-03-08 19:00:19,388 DEBUG [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1 2023-03-08 19:00:19,388 DEBUG [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:34283 2023-03-08 19:00:19,388 DEBUG [RS:2;jenkins-hbase6:46129] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:34283 2023-03-08 19:00:19,388 DEBUG [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-08 19:00:19,388 DEBUG [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:34283 2023-03-08 19:00:19,388 DEBUG [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-08 19:00:19,388 DEBUG [RS:2;jenkins-hbase6:46129] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-08 19:00:19,398 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-08 19:00:19,398 DEBUG [RS:1;jenkins-hbase6:41161] zookeeper.ZKUtil(162): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:19,398 DEBUG [RS:0;jenkins-hbase6:33875] zookeeper.ZKUtil(162): regionserver:33875-0x101d3eda4f90001, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:19,398 DEBUG [RS:2;jenkins-hbase6:46129] zookeeper.ZKUtil(162): regionserver:46129-0x101d3eda4f90003, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:19,398 WARN [RS:1;jenkins-hbase6:41161] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-08 19:00:19,399 WARN [RS:2;jenkins-hbase6:46129] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-08 19:00:19,399 INFO [RS:1;jenkins-hbase6:41161] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-08 19:00:19,398 WARN [RS:0;jenkins-hbase6:33875] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-08 19:00:19,399 INFO [RS:2;jenkins-hbase6:46129] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-08 19:00:19,400 DEBUG [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(1947): logDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/WALs/jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:19,400 DEBUG [RS:2;jenkins-hbase6:46129] regionserver.HRegionServer(1947): logDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/WALs/jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:19,400 INFO [RS:0;jenkins-hbase6:33875] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-08 19:00:19,400 DEBUG [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(1947): logDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/WALs/jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:19,400 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase6.apache.org,46129,1678302018069] 2023-03-08 19:00:19,400 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase6.apache.org,33875,1678302017939] 2023-03-08 19:00:19,400 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase6.apache.org,41161,1678302018011] 2023-03-08 19:00:19,413 DEBUG [RS:0;jenkins-hbase6:33875] zookeeper.ZKUtil(162): regionserver:33875-0x101d3eda4f90001, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:19,413 DEBUG [RS:2;jenkins-hbase6:46129] zookeeper.ZKUtil(162): regionserver:46129-0x101d3eda4f90003, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:19,413 DEBUG [RS:1;jenkins-hbase6:41161] zookeeper.ZKUtil(162): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:19,414 DEBUG [RS:0;jenkins-hbase6:33875] zookeeper.ZKUtil(162): regionserver:33875-0x101d3eda4f90001, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:19,414 DEBUG [RS:2;jenkins-hbase6:46129] zookeeper.ZKUtil(162): regionserver:46129-0x101d3eda4f90003, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:19,414 DEBUG [RS:0;jenkins-hbase6:33875] zookeeper.ZKUtil(162): regionserver:33875-0x101d3eda4f90001, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:19,414 DEBUG [RS:1;jenkins-hbase6:41161] zookeeper.ZKUtil(162): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:19,414 DEBUG [RS:2;jenkins-hbase6:46129] zookeeper.ZKUtil(162): regionserver:46129-0x101d3eda4f90003, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:19,415 DEBUG [RS:1;jenkins-hbase6:41161] zookeeper.ZKUtil(162): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:19,424 DEBUG [RS:0;jenkins-hbase6:33875] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-08 19:00:19,424 DEBUG [RS:2;jenkins-hbase6:46129] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-08 19:00:19,424 DEBUG [RS:1;jenkins-hbase6:41161] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-08 19:00:19,433 INFO [RS:0;jenkins-hbase6:33875] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-08 19:00:19,433 INFO [RS:2;jenkins-hbase6:46129] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-08 19:00:19,433 INFO [RS:1;jenkins-hbase6:41161] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-08 19:00:19,457 INFO [RS:2;jenkins-hbase6:46129] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-08 19:00:19,457 INFO [RS:1;jenkins-hbase6:41161] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-08 19:00:19,457 INFO [RS:0;jenkins-hbase6:33875] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-08 19:00:19,462 INFO [RS:0;jenkins-hbase6:33875] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-08 19:00:19,462 INFO [RS:2;jenkins-hbase6:46129] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-08 19:00:19,462 INFO [RS:0;jenkins-hbase6:33875] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,462 INFO [RS:1;jenkins-hbase6:41161] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-08 19:00:19,463 INFO [RS:2;jenkins-hbase6:46129] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,463 INFO [RS:1;jenkins-hbase6:41161] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,463 INFO [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-08 19:00:19,463 INFO [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-08 19:00:19,463 INFO [RS:2;jenkins-hbase6:46129] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-08 19:00:19,471 INFO [RS:2;jenkins-hbase6:46129] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,471 INFO [RS:0;jenkins-hbase6:33875] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,471 INFO [RS:1;jenkins-hbase6:41161] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,471 DEBUG [RS:2;jenkins-hbase6:46129] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,471 DEBUG [RS:0;jenkins-hbase6:33875] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,472 DEBUG [RS:2;jenkins-hbase6:46129] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,471 DEBUG [RS:1;jenkins-hbase6:41161] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,472 DEBUG [RS:2;jenkins-hbase6:46129] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,472 DEBUG [RS:1;jenkins-hbase6:41161] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,472 DEBUG [RS:2;jenkins-hbase6:46129] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,472 DEBUG [RS:1;jenkins-hbase6:41161] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,472 DEBUG [RS:2;jenkins-hbase6:46129] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,472 DEBUG [RS:1;jenkins-hbase6:41161] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,472 DEBUG [RS:2;jenkins-hbase6:46129] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase6:0, corePoolSize=2, maxPoolSize=2 2023-03-08 19:00:19,472 DEBUG [RS:1;jenkins-hbase6:41161] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,472 DEBUG [RS:2;jenkins-hbase6:46129] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,472 DEBUG [RS:1;jenkins-hbase6:41161] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase6:0, corePoolSize=2, maxPoolSize=2 2023-03-08 19:00:19,472 DEBUG [RS:0;jenkins-hbase6:33875] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,472 DEBUG [RS:1;jenkins-hbase6:41161] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,472 DEBUG [RS:2;jenkins-hbase6:46129] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,472 DEBUG [RS:1;jenkins-hbase6:41161] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,472 DEBUG [RS:2;jenkins-hbase6:46129] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,473 DEBUG [RS:1;jenkins-hbase6:41161] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,473 DEBUG [RS:2;jenkins-hbase6:46129] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,473 DEBUG [RS:1;jenkins-hbase6:41161] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,472 DEBUG [RS:0;jenkins-hbase6:33875] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,473 DEBUG [RS:0;jenkins-hbase6:33875] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,474 DEBUG [RS:0;jenkins-hbase6:33875] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,474 INFO [RS:2;jenkins-hbase6:46129] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,474 DEBUG [RS:0;jenkins-hbase6:33875] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase6:0, corePoolSize=2, maxPoolSize=2 2023-03-08 19:00:19,474 INFO [RS:2;jenkins-hbase6:46129] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,474 DEBUG [RS:0;jenkins-hbase6:33875] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,474 INFO [RS:2;jenkins-hbase6:46129] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,474 DEBUG [RS:0;jenkins-hbase6:33875] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,474 DEBUG [RS:0;jenkins-hbase6:33875] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,475 DEBUG [RS:0;jenkins-hbase6:33875] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-03-08 19:00:19,478 INFO [RS:1;jenkins-hbase6:41161] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,478 INFO [RS:1;jenkins-hbase6:41161] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,478 INFO [RS:1;jenkins-hbase6:41161] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,479 INFO [RS:0;jenkins-hbase6:33875] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,479 INFO [RS:0;jenkins-hbase6:33875] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,479 INFO [RS:0;jenkins-hbase6:33875] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,496 INFO [RS:2;jenkins-hbase6:46129] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-08 19:00:19,496 INFO [RS:0;jenkins-hbase6:33875] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-08 19:00:19,496 INFO [RS:1;jenkins-hbase6:41161] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-08 19:00:19,499 INFO [RS:2;jenkins-hbase6:46129] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase6.apache.org,46129,1678302018069-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,499 INFO [RS:0;jenkins-hbase6:33875] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase6.apache.org,33875,1678302017939-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,499 INFO [RS:1;jenkins-hbase6:41161] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase6.apache.org,41161,1678302018011-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:19,525 INFO [RS:0;jenkins-hbase6:33875] regionserver.Replication(203): jenkins-hbase6.apache.org,33875,1678302017939 started 2023-03-08 19:00:19,525 INFO [RS:1;jenkins-hbase6:41161] regionserver.Replication(203): jenkins-hbase6.apache.org,41161,1678302018011 started 2023-03-08 19:00:19,526 INFO [RS:2;jenkins-hbase6:46129] regionserver.Replication(203): jenkins-hbase6.apache.org,46129,1678302018069 started 2023-03-08 19:00:19,526 INFO [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(1638): Serving as jenkins-hbase6.apache.org,41161,1678302018011, RpcServer on jenkins-hbase6.apache.org/172.31.2.20:41161, sessionid=0x101d3eda4f90002 2023-03-08 19:00:19,525 INFO [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(1638): Serving as jenkins-hbase6.apache.org,33875,1678302017939, RpcServer on jenkins-hbase6.apache.org/172.31.2.20:33875, sessionid=0x101d3eda4f90001 2023-03-08 19:00:19,526 INFO [RS:2;jenkins-hbase6:46129] regionserver.HRegionServer(1638): Serving as jenkins-hbase6.apache.org,46129,1678302018069, RpcServer on jenkins-hbase6.apache.org/172.31.2.20:46129, sessionid=0x101d3eda4f90003 2023-03-08 19:00:19,526 DEBUG [RS:2;jenkins-hbase6:46129] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-08 19:00:19,526 DEBUG [RS:1;jenkins-hbase6:41161] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-08 19:00:19,526 DEBUG [RS:0;jenkins-hbase6:33875] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-08 19:00:19,526 DEBUG [RS:1;jenkins-hbase6:41161] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:19,526 DEBUG [RS:2;jenkins-hbase6:46129] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:19,527 DEBUG [RS:1;jenkins-hbase6:41161] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase6.apache.org,41161,1678302018011' 2023-03-08 19:00:19,527 DEBUG [RS:2;jenkins-hbase6:46129] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase6.apache.org,46129,1678302018069' 2023-03-08 19:00:19,528 DEBUG [RS:2;jenkins-hbase6:46129] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-08 19:00:19,527 DEBUG [RS:0;jenkins-hbase6:33875] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:19,528 DEBUG [RS:0;jenkins-hbase6:33875] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase6.apache.org,33875,1678302017939' 2023-03-08 19:00:19,528 DEBUG [RS:0;jenkins-hbase6:33875] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-08 19:00:19,528 DEBUG [RS:1;jenkins-hbase6:41161] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-08 19:00:19,528 DEBUG [RS:2;jenkins-hbase6:46129] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-08 19:00:19,529 DEBUG [RS:1;jenkins-hbase6:41161] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-08 19:00:19,529 DEBUG [RS:0;jenkins-hbase6:33875] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-08 19:00:19,529 DEBUG [RS:1;jenkins-hbase6:41161] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-08 19:00:19,529 DEBUG [RS:0;jenkins-hbase6:33875] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-08 19:00:19,529 DEBUG [RS:1;jenkins-hbase6:41161] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-08 19:00:19,529 DEBUG [RS:0;jenkins-hbase6:33875] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-08 19:00:19,530 DEBUG [RS:1;jenkins-hbase6:41161] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:19,530 DEBUG [RS:0;jenkins-hbase6:33875] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:19,531 DEBUG [RS:0;jenkins-hbase6:33875] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase6.apache.org,33875,1678302017939' 2023-03-08 19:00:19,531 DEBUG [RS:0;jenkins-hbase6:33875] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-08 19:00:19,530 DEBUG [RS:1;jenkins-hbase6:41161] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase6.apache.org,41161,1678302018011' 2023-03-08 19:00:19,531 DEBUG [RS:1;jenkins-hbase6:41161] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-08 19:00:19,531 DEBUG [RS:2;jenkins-hbase6:46129] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-08 19:00:19,531 DEBUG [RS:2;jenkins-hbase6:46129] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-08 19:00:19,531 DEBUG [RS:2;jenkins-hbase6:46129] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:19,531 DEBUG [RS:2;jenkins-hbase6:46129] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase6.apache.org,46129,1678302018069' 2023-03-08 19:00:19,531 DEBUG [RS:2;jenkins-hbase6:46129] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-08 19:00:19,531 DEBUG [RS:0;jenkins-hbase6:33875] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-08 19:00:19,532 DEBUG [RS:1;jenkins-hbase6:41161] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-08 19:00:19,532 DEBUG [RS:2;jenkins-hbase6:46129] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-08 19:00:19,532 DEBUG [RS:0;jenkins-hbase6:33875] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-08 19:00:19,532 INFO [RS:0;jenkins-hbase6:33875] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-08 19:00:19,532 INFO [RS:0;jenkins-hbase6:33875] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-08 19:00:19,534 DEBUG [RS:1;jenkins-hbase6:41161] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-08 19:00:19,534 DEBUG [RS:2;jenkins-hbase6:46129] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-08 19:00:19,534 INFO [RS:1;jenkins-hbase6:41161] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-08 19:00:19,535 INFO [RS:2;jenkins-hbase6:46129] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-08 19:00:19,535 INFO [RS:1;jenkins-hbase6:41161] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-08 19:00:19,536 INFO [RS:2;jenkins-hbase6:46129] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-08 19:00:19,538 DEBUG [jenkins-hbase6:43879] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-03-08 19:00:19,543 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase6.apache.org=0} racks are {/default-rack=0} 2023-03-08 19:00:19,549 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-08 19:00:19,549 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-03-08 19:00:19,549 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-03-08 19:00:19,550 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-03-08 19:00:19,552 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase6.apache.org,33875,1678302017939, state=OPENING 2023-03-08 19:00:19,559 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-08 19:00:19,560 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-08 19:00:19,561 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-08 19:00:19,564 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:19,643 INFO [RS:2;jenkins-hbase6:46129] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase6.apache.org%2C46129%2C1678302018069, suffix=, logDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/WALs/jenkins-hbase6.apache.org,46129,1678302018069, archiveDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/oldWALs, maxLogs=32 2023-03-08 19:00:19,643 INFO [RS:1;jenkins-hbase6:41161] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase6.apache.org%2C41161%2C1678302018011, suffix=, logDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/WALs/jenkins-hbase6.apache.org,41161,1678302018011, archiveDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/oldWALs, maxLogs=32 2023-03-08 19:00:19,645 INFO [RS:0;jenkins-hbase6:33875] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase6.apache.org%2C33875%2C1678302017939, suffix=, logDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/WALs/jenkins-hbase6.apache.org,33875,1678302017939, archiveDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/oldWALs, maxLogs=32 2023-03-08 19:00:19,672 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42929,DS-142238fa-90dc-4cd1-9438-7f1639f03bd7,DISK] 2023-03-08 19:00:19,672 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42379,DS-6d50f5f3-dee3-4200-a565-87ad0d2ea3ec,DISK] 2023-03-08 19:00:19,673 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34881,DS-9ddae0ae-a29a-46d0-ad70-eae1c0ede482,DISK] 2023-03-08 19:00:19,675 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42929,DS-142238fa-90dc-4cd1-9438-7f1639f03bd7,DISK] 2023-03-08 19:00:19,675 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42379,DS-6d50f5f3-dee3-4200-a565-87ad0d2ea3ec,DISK] 2023-03-08 19:00:19,684 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34881,DS-9ddae0ae-a29a-46d0-ad70-eae1c0ede482,DISK] 2023-03-08 19:00:19,687 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42379,DS-6d50f5f3-dee3-4200-a565-87ad0d2ea3ec,DISK] 2023-03-08 19:00:19,687 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42929,DS-142238fa-90dc-4cd1-9438-7f1639f03bd7,DISK] 2023-03-08 19:00:19,687 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34881,DS-9ddae0ae-a29a-46d0-ad70-eae1c0ede482,DISK] 2023-03-08 19:00:19,697 INFO [RS:1;jenkins-hbase6:41161] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/WALs/jenkins-hbase6.apache.org,41161,1678302018011/jenkins-hbase6.apache.org%2C41161%2C1678302018011.1678302019647 2023-03-08 19:00:19,697 INFO [RS:2;jenkins-hbase6:46129] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/WALs/jenkins-hbase6.apache.org,46129,1678302018069/jenkins-hbase6.apache.org%2C46129%2C1678302018069.1678302019648 2023-03-08 19:00:19,697 INFO [RS:0;jenkins-hbase6:33875] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/WALs/jenkins-hbase6.apache.org,33875,1678302017939/jenkins-hbase6.apache.org%2C33875%2C1678302017939.1678302019648 2023-03-08 19:00:19,698 DEBUG [RS:1;jenkins-hbase6:41161] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42929,DS-142238fa-90dc-4cd1-9438-7f1639f03bd7,DISK], DatanodeInfoWithStorage[127.0.0.1:34881,DS-9ddae0ae-a29a-46d0-ad70-eae1c0ede482,DISK], DatanodeInfoWithStorage[127.0.0.1:42379,DS-6d50f5f3-dee3-4200-a565-87ad0d2ea3ec,DISK]] 2023-03-08 19:00:19,698 DEBUG [RS:0;jenkins-hbase6:33875] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42929,DS-142238fa-90dc-4cd1-9438-7f1639f03bd7,DISK], DatanodeInfoWithStorage[127.0.0.1:42379,DS-6d50f5f3-dee3-4200-a565-87ad0d2ea3ec,DISK], DatanodeInfoWithStorage[127.0.0.1:34881,DS-9ddae0ae-a29a-46d0-ad70-eae1c0ede482,DISK]] 2023-03-08 19:00:19,699 DEBUG [RS:2;jenkins-hbase6:46129] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34881,DS-9ddae0ae-a29a-46d0-ad70-eae1c0ede482,DISK], DatanodeInfoWithStorage[127.0.0.1:42379,DS-6d50f5f3-dee3-4200-a565-87ad0d2ea3ec,DISK], DatanodeInfoWithStorage[127.0.0.1:42929,DS-142238fa-90dc-4cd1-9438-7f1639f03bd7,DISK]] 2023-03-08 19:00:19,748 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:19,750 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-08 19:00:19,753 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:36614, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-08 19:00:19,770 INFO [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-08 19:00:19,770 INFO [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-08 19:00:19,774 INFO [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase6.apache.org%2C33875%2C1678302017939.meta, suffix=.meta, logDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/WALs/jenkins-hbase6.apache.org,33875,1678302017939, archiveDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/oldWALs, maxLogs=32 2023-03-08 19:00:19,796 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42379,DS-6d50f5f3-dee3-4200-a565-87ad0d2ea3ec,DISK] 2023-03-08 19:00:19,798 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42929,DS-142238fa-90dc-4cd1-9438-7f1639f03bd7,DISK] 2023-03-08 19:00:19,798 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34881,DS-9ddae0ae-a29a-46d0-ad70-eae1c0ede482,DISK] 2023-03-08 19:00:19,804 INFO [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/WALs/jenkins-hbase6.apache.org,33875,1678302017939/jenkins-hbase6.apache.org%2C33875%2C1678302017939.meta.1678302019776.meta 2023-03-08 19:00:19,804 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42379,DS-6d50f5f3-dee3-4200-a565-87ad0d2ea3ec,DISK], DatanodeInfoWithStorage[127.0.0.1:34881,DS-9ddae0ae-a29a-46d0-ad70-eae1c0ede482,DISK], DatanodeInfoWithStorage[127.0.0.1:42929,DS-142238fa-90dc-4cd1-9438-7f1639f03bd7,DISK]] 2023-03-08 19:00:19,805 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-08 19:00:19,807 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-08 19:00:19,827 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-08 19:00:19,832 INFO [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-08 19:00:19,837 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-08 19:00:19,838 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:19,838 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-08 19:00:19,838 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-08 19:00:19,841 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-08 19:00:19,842 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/info 2023-03-08 19:00:19,843 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/info 2023-03-08 19:00:19,843 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-08 19:00:19,844 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:19,844 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-08 19:00:19,846 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/rep_barrier 2023-03-08 19:00:19,846 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/rep_barrier 2023-03-08 19:00:19,846 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-08 19:00:19,847 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:19,847 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-08 19:00:19,849 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/table 2023-03-08 19:00:19,849 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/table 2023-03-08 19:00:19,849 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-08 19:00:19,850 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:19,852 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740 2023-03-08 19:00:19,854 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740 2023-03-08 19:00:19,858 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-03-08 19:00:19,860 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-08 19:00:19,861 INFO [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66437609, jitterRate=-0.010002478957176208}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-03-08 19:00:19,862 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-08 19:00:19,871 INFO [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1678302019740 2023-03-08 19:00:19,888 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-08 19:00:19,889 INFO [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-08 19:00:19,889 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase6.apache.org,33875,1678302017939, state=OPEN 2023-03-08 19:00:19,892 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-08 19:00:19,892 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-08 19:00:19,897 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-08 19:00:19,897 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase6.apache.org,33875,1678302017939 in 328 msec 2023-03-08 19:00:19,902 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-08 19:00:19,902 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 537 msec 2023-03-08 19:00:19,908 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 755 msec 2023-03-08 19:00:19,908 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1678302019908, completionTime=-1 2023-03-08 19:00:19,908 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-03-08 19:00:19,908 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-03-08 19:00:19,973 DEBUG [hconnection-0x6f162865-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:19,976 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:36616, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:19,992 INFO [master/jenkins-hbase6:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-03-08 19:00:19,992 INFO [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1678302079992 2023-03-08 19:00:19,992 INFO [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1678302139992 2023-03-08 19:00:19,992 INFO [master/jenkins-hbase6:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 83 msec 2023-03-08 19:00:20,016 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase6.apache.org,43879,1678302016597-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:20,017 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase6.apache.org,43879,1678302016597-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:20,017 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase6.apache.org,43879,1678302016597-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:20,019 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase6:43879, period=300000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:20,019 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-08 19:00:20,027 DEBUG [master/jenkins-hbase6:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-08 19:00:20,039 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-08 19:00:20,040 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-08 19:00:20,053 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-08 19:00:20,055 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-08 19:00:20,062 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-08 19:00:20,086 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/hbase/namespace/ff6757431e14fac2c9bca54634071a1a 2023-03-08 19:00:20,088 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/hbase/namespace/ff6757431e14fac2c9bca54634071a1a empty. 2023-03-08 19:00:20,089 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/hbase/namespace/ff6757431e14fac2c9bca54634071a1a 2023-03-08 19:00:20,089 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-08 19:00:20,176 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-08 19:00:20,179 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => ff6757431e14fac2c9bca54634071a1a, NAME => 'hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:20,197 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:20,197 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing ff6757431e14fac2c9bca54634071a1a, disabling compactions & flushes 2023-03-08 19:00:20,197 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a. 2023-03-08 19:00:20,197 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a. 2023-03-08 19:00:20,197 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a. after waiting 0 ms 2023-03-08 19:00:20,197 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a. 2023-03-08 19:00:20,197 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a. 2023-03-08 19:00:20,197 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for ff6757431e14fac2c9bca54634071a1a: 2023-03-08 19:00:20,202 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-08 19:00:20,217 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1678302020204"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302020204"}]},"ts":"1678302020204"} 2023-03-08 19:00:20,244 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-08 19:00:20,246 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-08 19:00:20,252 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302020246"}]},"ts":"1678302020246"} 2023-03-08 19:00:20,262 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-08 19:00:20,266 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase6.apache.org=0} racks are {/default-rack=0} 2023-03-08 19:00:20,267 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-08 19:00:20,268 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-03-08 19:00:20,268 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-03-08 19:00:20,268 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-03-08 19:00:20,270 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=ff6757431e14fac2c9bca54634071a1a, ASSIGN}] 2023-03-08 19:00:20,273 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=ff6757431e14fac2c9bca54634071a1a, ASSIGN 2023-03-08 19:00:20,274 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=ff6757431e14fac2c9bca54634071a1a, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,41161,1678302018011; forceNewPlan=false, retain=false 2023-03-08 19:00:20,427 INFO [jenkins-hbase6:43879] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-03-08 19:00:20,428 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=ff6757431e14fac2c9bca54634071a1a, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:20,428 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1678302020428"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302020428"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302020428"}]},"ts":"1678302020428"} 2023-03-08 19:00:20,433 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure ff6757431e14fac2c9bca54634071a1a, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:20,588 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:20,588 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-08 19:00:20,592 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:35306, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-08 19:00:20,598 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a. 2023-03-08 19:00:20,598 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ff6757431e14fac2c9bca54634071a1a, NAME => 'hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a.', STARTKEY => '', ENDKEY => ''} 2023-03-08 19:00:20,599 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace ff6757431e14fac2c9bca54634071a1a 2023-03-08 19:00:20,599 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:20,600 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for ff6757431e14fac2c9bca54634071a1a 2023-03-08 19:00:20,600 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for ff6757431e14fac2c9bca54634071a1a 2023-03-08 19:00:20,602 INFO [StoreOpener-ff6757431e14fac2c9bca54634071a1a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region ff6757431e14fac2c9bca54634071a1a 2023-03-08 19:00:20,604 DEBUG [StoreOpener-ff6757431e14fac2c9bca54634071a1a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/namespace/ff6757431e14fac2c9bca54634071a1a/info 2023-03-08 19:00:20,604 DEBUG [StoreOpener-ff6757431e14fac2c9bca54634071a1a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/namespace/ff6757431e14fac2c9bca54634071a1a/info 2023-03-08 19:00:20,605 INFO [StoreOpener-ff6757431e14fac2c9bca54634071a1a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ff6757431e14fac2c9bca54634071a1a columnFamilyName info 2023-03-08 19:00:20,606 INFO [StoreOpener-ff6757431e14fac2c9bca54634071a1a-1] regionserver.HStore(310): Store=ff6757431e14fac2c9bca54634071a1a/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:20,608 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/namespace/ff6757431e14fac2c9bca54634071a1a 2023-03-08 19:00:20,609 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/namespace/ff6757431e14fac2c9bca54634071a1a 2023-03-08 19:00:20,614 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for ff6757431e14fac2c9bca54634071a1a 2023-03-08 19:00:20,620 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/namespace/ff6757431e14fac2c9bca54634071a1a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-08 19:00:20,620 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened ff6757431e14fac2c9bca54634071a1a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70351618, jitterRate=0.04832080006599426}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:20,621 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for ff6757431e14fac2c9bca54634071a1a: 2023-03-08 19:00:20,623 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a., pid=6, masterSystemTime=1678302020588 2023-03-08 19:00:20,628 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a. 2023-03-08 19:00:20,628 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a. 2023-03-08 19:00:20,630 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=ff6757431e14fac2c9bca54634071a1a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:20,630 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1678302020629"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302020629"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302020629"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302020629"}]},"ts":"1678302020629"} 2023-03-08 19:00:20,637 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-08 19:00:20,638 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure ff6757431e14fac2c9bca54634071a1a, server=jenkins-hbase6.apache.org,41161,1678302018011 in 200 msec 2023-03-08 19:00:20,641 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-08 19:00:20,642 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=ff6757431e14fac2c9bca54634071a1a, ASSIGN in 368 msec 2023-03-08 19:00:20,643 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-08 19:00:20,644 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302020643"}]},"ts":"1678302020643"} 2023-03-08 19:00:20,646 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-08 19:00:20,649 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-08 19:00:20,652 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 608 msec 2023-03-08 19:00:20,660 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-08 19:00:20,661 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-08 19:00:20,661 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-08 19:00:20,685 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:20,689 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:35320, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:20,706 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-08 19:00:20,726 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-08 19:00:20,734 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 39 msec 2023-03-08 19:00:20,740 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-08 19:00:20,752 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-08 19:00:20,757 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-03-08 19:00:20,767 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-08 19:00:20,769 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-08 19:00:20,769 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.633sec 2023-03-08 19:00:20,772 INFO [master/jenkins-hbase6:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-08 19:00:20,774 INFO [master/jenkins-hbase6:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-08 19:00:20,774 INFO [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-08 19:00:20,775 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase6.apache.org,43879,1678302016597-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-08 19:00:20,777 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase6.apache.org,43879,1678302016597-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-08 19:00:20,786 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-08 19:00:20,801 DEBUG [Listener at localhost/45021] zookeeper.ReadOnlyZKClient(139): Connect 0x492b920e to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:20,811 DEBUG [Listener at localhost/45021] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c4d98c3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:20,828 DEBUG [hconnection-0x3fa82af-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:20,844 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:36618, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:20,854 INFO [Listener at localhost/45021] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase6.apache.org,43879,1678302016597 2023-03-08 19:00:20,887 INFO [Listener at localhost/45021] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=362, OpenFileDescriptor=695, MaxFileDescriptor=60000, SystemLoadAverage=391, ProcessCount=169, AvailableMemoryMB=7283 2023-03-08 19:00:20,902 DEBUG [Listener at localhost/45021] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-08 19:00:20,906 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:36330, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-08 19:00:20,922 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$4(2112): Client=jenkins//172.31.2.20 create 'testRestoreSnapshot-1678302020890', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-03-08 19:00:20,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:20,927 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1678302020890 execute state=CREATE_TABLE_PRE_OPERATION 2023-03-08 19:00:20,930 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot-1678302020890 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-08 19:00:20,932 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(697): Client=jenkins//172.31.2.20 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot-1678302020890" procId is: 9 2023-03-08 19:00:20,939 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:20,939 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:20,939 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:20,940 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:20,940 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 empty. 2023-03-08 19:00:20,940 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:20,940 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:20,940 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 empty. 2023-03-08 19:00:20,940 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 empty. 2023-03-08 19:00:20,941 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d empty. 2023-03-08 19:00:20,943 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd empty. 2023-03-08 19:00:20,943 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:20,944 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 empty. 2023-03-08 19:00:20,944 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:20,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-08 19:00:20,946 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:20,947 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:20,947 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:20,947 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:20,947 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1678302020890 regions 2023-03-08 19:00:20,972 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/.tabledesc/.tableinfo.0000000001 2023-03-08 19:00:20,974 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(7675): creating {ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1678302020890', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:20,974 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(7675): creating {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1678302020890', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:20,975 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(7675): creating {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1678302020890', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:21,025 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:21,026 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(1604): Closing be16768f5782c50d4c94d5fdc74d3f28, disabling compactions & flushes 2023-03-08 19:00:21,026 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:21,027 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:21,027 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. after waiting 0 ms 2023-03-08 19:00:21,027 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:21,027 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:21,027 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(1558): Region close journal for be16768f5782c50d4c94d5fdc74d3f28: 2023-03-08 19:00:21,028 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(7675): creating {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1678302020890', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:21,029 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:21,029 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:21,030 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(1604): Closing 668bd868cc8589cf26791b7dda1212d8, disabling compactions & flushes 2023-03-08 19:00:21,030 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(1604): Closing 7cdf03195577655a4e5d1e7f4ff92392, disabling compactions & flushes 2023-03-08 19:00:21,030 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:21,030 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:21,030 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:21,030 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:21,030 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. after waiting 0 ms 2023-03-08 19:00:21,030 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. after waiting 0 ms 2023-03-08 19:00:21,030 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:21,030 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:21,031 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:21,031 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:21,031 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(1558): Region close journal for 668bd868cc8589cf26791b7dda1212d8: 2023-03-08 19:00:21,031 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(1558): Region close journal for 7cdf03195577655a4e5d1e7f4ff92392: 2023-03-08 19:00:21,031 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(7675): creating {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1678302020890', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:21,032 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(7675): creating {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1678302020890', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:21,076 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:21,077 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(1604): Closing b8add49535b4b0366d051f3a48b2a333, disabling compactions & flushes 2023-03-08 19:00:21,077 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:21,077 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:21,077 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. after waiting 0 ms 2023-03-08 19:00:21,077 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:21,077 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:21,077 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-1] regionserver.HRegion(1558): Region close journal for b8add49535b4b0366d051f3a48b2a333: 2023-03-08 19:00:21,078 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:21,078 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:21,078 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(1604): Closing ea43d1afb394fd7a05814d2275bdeffd, disabling compactions & flushes 2023-03-08 19:00:21,079 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(1604): Closing e3eaed3ac05b4e65948c1d4b5c75462d, disabling compactions & flushes 2023-03-08 19:00:21,079 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:21,079 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:21,079 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. after waiting 0 ms 2023-03-08 19:00:21,079 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:21,079 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:21,079 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-0] regionserver.HRegion(1558): Region close journal for ea43d1afb394fd7a05814d2275bdeffd: 2023-03-08 19:00:21,079 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:21,079 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:21,079 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. after waiting 0 ms 2023-03-08 19:00:21,079 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:21,079 INFO [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:21,079 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1678302020890-pool-2] regionserver.HRegion(1558): Region close journal for e3eaed3ac05b4e65948c1d4b5c75462d: 2023-03-08 19:00:21,085 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot-1678302020890 execute state=CREATE_TABLE_ADD_TO_META 2023-03-08 19:00:21,087 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021086"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302021086"}]},"ts":"1678302021086"} 2023-03-08 19:00:21,087 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302021086"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302021086"}]},"ts":"1678302021086"} 2023-03-08 19:00:21,087 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021086"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302021086"}]},"ts":"1678302021086"} 2023-03-08 19:00:21,088 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021086"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302021086"}]},"ts":"1678302021086"} 2023-03-08 19:00:21,088 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021086"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302021086"}]},"ts":"1678302021086"} 2023-03-08 19:00:21,088 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302021086"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302021086"}]},"ts":"1678302021086"} 2023-03-08 19:00:21,143 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-03-08 19:00:21,145 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot-1678302020890 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-08 19:00:21,146 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302021145"}]},"ts":"1678302021145"} 2023-03-08 19:00:21,148 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=ENABLING in hbase:meta 2023-03-08 19:00:21,153 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase6.apache.org=0} racks are {/default-rack=0} 2023-03-08 19:00:21,159 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-08 19:00:21,159 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-03-08 19:00:21,159 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-03-08 19:00:21,159 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-03-08 19:00:21,159 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN}] 2023-03-08 19:00:21,163 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN 2023-03-08 19:00:21,164 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN 2023-03-08 19:00:21,164 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN 2023-03-08 19:00:21,165 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN 2023-03-08 19:00:21,167 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,41161,1678302018011; forceNewPlan=false, retain=false 2023-03-08 19:00:21,167 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,33875,1678302017939; forceNewPlan=false, retain=false 2023-03-08 19:00:21,167 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,46129,1678302018069; forceNewPlan=false, retain=false 2023-03-08 19:00:21,167 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,46129,1678302018069; forceNewPlan=false, retain=false 2023-03-08 19:00:21,168 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN 2023-03-08 19:00:21,168 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN 2023-03-08 19:00:21,169 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,41161,1678302018011; forceNewPlan=false, retain=false 2023-03-08 19:00:21,170 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,33875,1678302017939; forceNewPlan=false, retain=false 2023-03-08 19:00:21,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-08 19:00:21,317 INFO [jenkins-hbase6:43879] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-03-08 19:00:21,320 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:21,320 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:21,320 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:21,320 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021320"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302021320"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302021320"}]},"ts":"1678302021320"} 2023-03-08 19:00:21,320 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302021320"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302021320"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302021320"}]},"ts":"1678302021320"} 2023-03-08 19:00:21,320 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:21,320 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021320"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302021320"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302021320"}]},"ts":"1678302021320"} 2023-03-08 19:00:21,320 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:21,321 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302021320"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302021320"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302021320"}]},"ts":"1678302021320"} 2023-03-08 19:00:21,321 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021320"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302021320"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302021320"}]},"ts":"1678302021320"} 2023-03-08 19:00:21,323 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=11, state=RUNNABLE; OpenRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:21,325 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=10, state=RUNNABLE; OpenRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:21,328 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=13, state=RUNNABLE; OpenRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:21,333 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:21,334 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=15, state=RUNNABLE; OpenRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:21,334 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021333"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302021333"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302021333"}]},"ts":"1678302021333"} 2023-03-08 19:00:21,336 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=12, state=RUNNABLE; OpenRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:21,338 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=14, state=RUNNABLE; OpenRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:21,477 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:21,477 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-08 19:00:21,479 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:34072, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-08 19:00:21,486 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:21,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'} 2023-03-08 19:00:21,487 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:21,487 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:21,487 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:21,487 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:21,488 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:21,488 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'} 2023-03-08 19:00:21,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:21,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:21,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:21,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:21,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:21,501 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:21,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''} 2023-03-08 19:00:21,501 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:21,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:21,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:21,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:21,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:21,507 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf 2023-03-08 19:00:21,507 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf 2023-03-08 19:00:21,507 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf 2023-03-08 19:00:21,507 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf 2023-03-08 19:00:21,511 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 668bd868cc8589cf26791b7dda1212d8 columnFamilyName cf 2023-03-08 19:00:21,511 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b8add49535b4b0366d051f3a48b2a333 columnFamilyName cf 2023-03-08 19:00:21,513 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:21,516 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(310): Store=b8add49535b4b0366d051f3a48b2a333/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:21,516 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(310): Store=668bd868cc8589cf26791b7dda1212d8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:21,519 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:21,519 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:21,520 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:21,520 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:21,522 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf 2023-03-08 19:00:21,522 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf 2023-03-08 19:00:21,523 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e3eaed3ac05b4e65948c1d4b5c75462d columnFamilyName cf 2023-03-08 19:00:21,530 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:21,532 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:21,533 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(310): Store=e3eaed3ac05b4e65948c1d4b5c75462d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:21,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:21,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-08 19:00:21,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-08 19:00:21,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:21,552 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened b8add49535b4b0366d051f3a48b2a333; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73110790, jitterRate=0.08943566679954529}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:21,552 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 668bd868cc8589cf26791b7dda1212d8; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72522671, jitterRate=0.0806720107793808}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:21,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for b8add49535b4b0366d051f3a48b2a333: 2023-03-08 19:00:21,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 668bd868cc8589cf26791b7dda1212d8: 2023-03-08 19:00:21,554 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333., pid=18, masterSystemTime=1678302021477 2023-03-08 19:00:21,556 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8., pid=17, masterSystemTime=1678302021481 2023-03-08 19:00:21,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:21,559 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:21,559 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:21,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'} 2023-03-08 19:00:21,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:21,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:21,561 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:21,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:21,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:21,561 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021560"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302021560"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302021560"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302021560"}]},"ts":"1678302021560"} 2023-03-08 19:00:21,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:21,561 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:21,561 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:21,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'} 2023-03-08 19:00:21,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:21,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:21,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:21,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:21,562 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:21,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:21,564 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302021562"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302021562"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302021562"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302021562"}]},"ts":"1678302021562"} 2023-03-08 19:00:21,571 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=13 2023-03-08 19:00:21,571 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=13, state=SUCCESS; OpenRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,46129,1678302018069 in 237 msec 2023-03-08 19:00:21,573 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:21,579 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:21,574 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=10 2023-03-08 19:00:21,579 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=10, state=SUCCESS; OpenRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939 in 243 msec 2023-03-08 19:00:21,580 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN in 412 msec 2023-03-08 19:00:21,580 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-08 19:00:21,582 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened e3eaed3ac05b4e65948c1d4b5c75462d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60971933, jitterRate=-0.09144739806652069}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:21,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for e3eaed3ac05b4e65948c1d4b5c75462d: 2023-03-08 19:00:21,584 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf 2023-03-08 19:00:21,584 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf 2023-03-08 19:00:21,584 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region be16768f5782c50d4c94d5fdc74d3f28 columnFamilyName cf 2023-03-08 19:00:21,585 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf 2023-03-08 19:00:21,585 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf 2023-03-08 19:00:21,586 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d., pid=19, masterSystemTime=1678302021487 2023-03-08 19:00:21,587 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ea43d1afb394fd7a05814d2275bdeffd columnFamilyName cf 2023-03-08 19:00:21,589 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(310): Store=ea43d1afb394fd7a05814d2275bdeffd/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:21,589 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(310): Store=be16768f5782c50d4c94d5fdc74d3f28/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:21,592 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN in 415 msec 2023-03-08 19:00:21,593 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:21,593 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:21,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:21,594 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:21,594 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:21,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'} 2023-03-08 19:00:21,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:21,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:21,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:21,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:21,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:21,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:21,596 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:21,597 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302021596"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302021596"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302021596"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302021596"}]},"ts":"1678302021596"} 2023-03-08 19:00:21,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:21,602 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:21,603 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:21,607 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=15 2023-03-08 19:00:21,607 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-08 19:00:21,607 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=15, state=SUCCESS; OpenRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 267 msec 2023-03-08 19:00:21,608 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf 2023-03-08 19:00:21,608 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf 2023-03-08 19:00:21,608 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened be16768f5782c50d4c94d5fdc74d3f28; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71350948, jitterRate=0.06321197748184204}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:21,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-08 19:00:21,609 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7cdf03195577655a4e5d1e7f4ff92392 columnFamilyName cf 2023-03-08 19:00:21,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for be16768f5782c50d4c94d5fdc74d3f28: 2023-03-08 19:00:21,610 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(310): Store=7cdf03195577655a4e5d1e7f4ff92392/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:21,612 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened ea43d1afb394fd7a05814d2275bdeffd; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63777062, jitterRate=-0.04964771866798401}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:21,612 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28., pid=16, masterSystemTime=1678302021477 2023-03-08 19:00:21,612 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for ea43d1afb394fd7a05814d2275bdeffd: 2023-03-08 19:00:21,613 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN in 448 msec 2023-03-08 19:00:21,613 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd., pid=21, masterSystemTime=1678302021481 2023-03-08 19:00:21,614 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:21,615 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:21,616 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:21,617 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021616"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302021616"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302021616"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302021616"}]},"ts":"1678302021616"} 2023-03-08 19:00:21,617 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:21,617 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:21,618 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:21,619 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:21,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:21,620 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021619"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302021619"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302021619"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302021619"}]},"ts":"1678302021619"} 2023-03-08 19:00:21,628 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=11 2023-03-08 19:00:21,629 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=11, state=SUCCESS; OpenRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,46129,1678302018069 in 301 msec 2023-03-08 19:00:21,629 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:21,632 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=14 2023-03-08 19:00:21,633 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN in 470 msec 2023-03-08 19:00:21,633 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=14, state=SUCCESS; OpenRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,33875,1678302017939 in 289 msec 2023-03-08 19:00:21,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-08 19:00:21,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 7cdf03195577655a4e5d1e7f4ff92392; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74224916, jitterRate=0.106037437915802}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:21,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 7cdf03195577655a4e5d1e7f4ff92392: 2023-03-08 19:00:21,635 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN in 474 msec 2023-03-08 19:00:21,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392., pid=20, masterSystemTime=1678302021487 2023-03-08 19:00:21,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:21,640 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:21,641 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:21,641 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021641"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302021641"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302021641"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302021641"}]},"ts":"1678302021641"} 2023-03-08 19:00:21,649 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=12 2023-03-08 19:00:21,649 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=12, state=SUCCESS; OpenRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,41161,1678302018011 in 308 msec 2023-03-08 19:00:21,658 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=12, resume processing ppid=9 2023-03-08 19:00:21,660 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN in 490 msec 2023-03-08 19:00:21,661 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot-1678302020890 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-08 19:00:21,661 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302021661"}]},"ts":"1678302021661"} 2023-03-08 19:00:21,663 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=ENABLED in hbase:meta 2023-03-08 19:00:21,666 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1678302020890 execute state=CREATE_TABLE_POST_OPERATION 2023-03-08 19:00:21,669 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot-1678302020890 in 744 msec 2023-03-08 19:00:21,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-08 19:00:21,702 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot-1678302020890, procId: 9 completed 2023-03-08 19:00:21,709 DEBUG [Listener at localhost/45021] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1678302020890 2023-03-08 19:00:21,711 DEBUG [Listener at localhost/45021] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:21,714 INFO [Listener at localhost/45021] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-03-08 19:00:21,741 DEBUG [hconnection-0x3fa82af-shared-pool-5] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:21,745 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:34074, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:21,746 DEBUG [hconnection-0x3fa82af-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:21,749 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:35330, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:21,761 INFO [Listener at localhost/45021] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1678302020890 2023-03-08 19:00:21,766 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$11(2418): Client=jenkins//172.31.2.20 disable testRestoreSnapshot-1678302020890 2023-03-08 19:00:21,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:21,778 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302021778"}]},"ts":"1678302021778"} 2023-03-08 19:00:21,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-03-08 19:00:21,781 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=DISABLING in hbase:meta 2023-03-08 19:00:21,783 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1678302020890 to state=DISABLING 2023-03-08 19:00:21,785 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN}, {pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN}, {pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN}, {pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN}, {pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN}, {pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN}] 2023-03-08 19:00:21,789 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN 2023-03-08 19:00:21,789 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN 2023-03-08 19:00:21,790 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN 2023-03-08 19:00:21,790 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN 2023-03-08 19:00:21,791 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN 2023-03-08 19:00:21,793 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:21,793 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:21,793 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302021793"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302021793"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302021793"}]},"ts":"1678302021793"} 2023-03-08 19:00:21,794 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:21,794 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:21,794 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302021793"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302021793"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302021793"}]},"ts":"1678302021793"} 2023-03-08 19:00:21,794 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021794"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302021794"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302021794"}]},"ts":"1678302021794"} 2023-03-08 19:00:21,793 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021793"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302021793"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302021793"}]},"ts":"1678302021793"} 2023-03-08 19:00:21,794 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:21,794 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021793"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302021793"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302021793"}]},"ts":"1678302021793"} 2023-03-08 19:00:21,796 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE; CloseRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:21,798 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=23, state=RUNNABLE; CloseRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:21,802 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=27, state=RUNNABLE; CloseRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:21,804 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=24, state=RUNNABLE; CloseRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:21,805 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=26, state=RUNNABLE; CloseRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:21,811 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN 2023-03-08 19:00:21,813 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:21,813 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021813"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302021813"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302021813"}]},"ts":"1678302021813"} 2023-03-08 19:00:21,816 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=25, state=RUNNABLE; CloseRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:21,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:21,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:21,962 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing ea43d1afb394fd7a05814d2275bdeffd, disabling compactions & flushes 2023-03-08 19:00:21,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 7cdf03195577655a4e5d1e7f4ff92392, disabling compactions & flushes 2023-03-08 19:00:21,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:21,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:21,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:21,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. after waiting 0 ms 2023-03-08 19:00:21,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:21,964 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:21,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. after waiting 0 ms 2023-03-08 19:00:21,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:21,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing b8add49535b4b0366d051f3a48b2a333, disabling compactions & flushes 2023-03-08 19:00:21,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:21,965 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:21,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:21,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. after waiting 0 ms 2023-03-08 19:00:21,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:21,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-08 19:00:21,974 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-08 19:00:21,974 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:21,974 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 7cdf03195577655a4e5d1e7f4ff92392: 2023-03-08 19:00:21,974 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:21,974 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for ea43d1afb394fd7a05814d2275bdeffd: 2023-03-08 19:00:21,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-08 19:00:21,976 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:21,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for b8add49535b4b0366d051f3a48b2a333: 2023-03-08 19:00:21,978 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:21,978 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:21,979 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing e3eaed3ac05b4e65948c1d4b5c75462d, disabling compactions & flushes 2023-03-08 19:00:21,979 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:21,979 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:21,979 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. after waiting 0 ms 2023-03-08 19:00:21,979 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:21,979 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=CLOSED 2023-03-08 19:00:21,980 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021979"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302021979"}]},"ts":"1678302021979"} 2023-03-08 19:00:21,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:21,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:21,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 668bd868cc8589cf26791b7dda1212d8, disabling compactions & flushes 2023-03-08 19:00:21,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:21,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:21,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. after waiting 0 ms 2023-03-08 19:00:21,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:21,985 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=CLOSED 2023-03-08 19:00:21,985 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021985"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302021985"}]},"ts":"1678302021985"} 2023-03-08 19:00:21,987 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:21,987 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:21,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing be16768f5782c50d4c94d5fdc74d3f28, disabling compactions & flushes 2023-03-08 19:00:21,988 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:21,989 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=CLOSED 2023-03-08 19:00:21,989 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302021989"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302021989"}]},"ts":"1678302021989"} 2023-03-08 19:00:21,991 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:21,991 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. after waiting 0 ms 2023-03-08 19:00:21,991 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:21,992 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=25 2023-03-08 19:00:21,993 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=25, state=SUCCESS; CloseRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,41161,1678302018011 in 171 msec 2023-03-08 19:00:21,994 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=27 2023-03-08 19:00:21,994 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=27, state=SUCCESS; CloseRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,33875,1678302017939 in 187 msec 2023-03-08 19:00:21,999 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN in 208 msec 2023-03-08 19:00:21,999 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN in 209 msec 2023-03-08 19:00:21,999 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=26 2023-03-08 19:00:21,999 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=26, state=SUCCESS; CloseRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,46129,1678302018069 in 188 msec 2023-03-08 19:00:22,002 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN in 214 msec 2023-03-08 19:00:22,008 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-08 19:00:22,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-08 19:00:22,009 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:22,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for e3eaed3ac05b4e65948c1d4b5c75462d: 2023-03-08 19:00:22,010 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:22,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 668bd868cc8589cf26791b7dda1212d8: 2023-03-08 19:00:22,013 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:22,014 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-08 19:00:22,014 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=CLOSED 2023-03-08 19:00:22,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:22,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for be16768f5782c50d4c94d5fdc74d3f28: 2023-03-08 19:00:22,015 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302022014"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302022014"}]},"ts":"1678302022014"} 2023-03-08 19:00:22,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:22,016 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=CLOSED 2023-03-08 19:00:22,016 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302022016"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302022016"}]},"ts":"1678302022016"} 2023-03-08 19:00:22,018 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:22,019 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=CLOSED 2023-03-08 19:00:22,019 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302022019"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302022019"}]},"ts":"1678302022019"} 2023-03-08 19:00:22,022 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-08 19:00:22,022 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; CloseRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 222 msec 2023-03-08 19:00:22,024 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=23 2023-03-08 19:00:22,024 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=23, state=SUCCESS; CloseRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939 in 221 msec 2023-03-08 19:00:22,026 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN in 237 msec 2023-03-08 19:00:22,026 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=24 2023-03-08 19:00:22,026 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=24, state=SUCCESS; CloseRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,46129,1678302018069 in 217 msec 2023-03-08 19:00:22,027 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN in 239 msec 2023-03-08 19:00:22,029 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=22 2023-03-08 19:00:22,029 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN in 241 msec 2023-03-08 19:00:22,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-03-08 19:00:22,032 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302022032"}]},"ts":"1678302022032"} 2023-03-08 19:00:22,034 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=DISABLED in hbase:meta 2023-03-08 19:00:22,039 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1678302020890 to state=DISABLED 2023-03-08 19:00:22,050 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1678302020890 in 273 msec 2023-03-08 19:00:22,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-03-08 19:00:22,534 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1678302020890, procId: 22 completed 2023-03-08 19:00:22,548 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1580): Client=jenkins//172.31.2.20 snapshot request for:{ ss=emptySnaptb-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:22,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1678302022548 (current time:1678302022548). 2023-03-08 19:00:22,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-03-08 19:00:22,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] zookeeper.ReadOnlyZKClient(139): Connect 0x409f0101 to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:22,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71df95c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:22,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:22,567 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:36628, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:22,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x409f0101 to 127.0.0.1:59169 2023-03-08 19:00:22,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:22,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-03-08 19:00:22,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:22,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:22,592 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:22,593 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=35, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE; timeout=600000, timestamp=1678302622593 2023-03-08 19:00:22,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:22,593 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1678302020890 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1678302020890 2023-03-08 19:00:22,598 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-03-08 19:00:22,598 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-03-08 19:00:22,611 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1678302020890 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } 2023-03-08 19:00:22,615 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:22,615 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:22,615 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:22,615 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:22,615 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:22,615 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:22,615 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:22,620 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:22,620 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:22,620 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:22,620 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:22,620 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:22,620 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:22,620 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:22,627 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:22,627 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:22,627 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:22,629 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:22,631 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:22,632 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:22,635 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:22,715 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1678302020890 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } as finished. 2023-03-08 19:00:22,715 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1678302020890 2023-03-08 19:00:22,717 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/emptySnaptb-1678302020890 2023-03-08 19:00:22,772 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/emptySnaptb-1678302020890 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/emptySnaptb-1678302020890 2023-03-08 19:00:22,813 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1678302020890 of table testRestoreSnapshot-1678302020890 completed 2023-03-08 19:00:22,813 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/emptySnaptb-1678302020890 2023-03-08 19:00:22,814 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/emptySnaptb-1678302020890 2023-03-08 19:00:22,814 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1678302020890 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1678302020890 at 1678302022593Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1678302020890 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } at 1678302022611 (+18 ms)Consolidate snapshot: emptySnaptb-1678302020890 at 1678302022715 (+104 ms)Loading Region manifests for emptySnaptb-1678302020890 at 1678302022715Writing data manifest for emptySnaptb-1678302020890 at 1678302022734 (+19 ms)Verifying snapshot: emptySnaptb-1678302020890 at 1678302022756 (+22 ms)Snapshot emptySnaptb-1678302020890 of table testRestoreSnapshot-1678302020890 completed at 1678302022813 (+57 ms) 2023-03-08 19:00:22,815 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=35, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:22,818 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE in 232 msec 2023-03-08 19:00:22,848 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-03-08 19:00:22,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } is done 2023-03-08 19:00:22,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 }' has completed, notifying client. 2023-03-08 19:00:22,856 INFO [Listener at localhost/45021] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1678302020890 2023-03-08 19:00:22,860 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$10(2385): Client=jenkins//172.31.2.20 enable testRestoreSnapshot-1678302020890 2023-03-08 19:00:22,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=36, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:22,874 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1678302020890 2023-03-08 19:00:22,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-03-08 19:00:22,874 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302022874"}]},"ts":"1678302022874"} 2023-03-08 19:00:22,876 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=ENABLING in hbase:meta 2023-03-08 19:00:22,880 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-03-08 19:00:22,882 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN}, {pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN}, {pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN}, {pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN}, {pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN}, {pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN}] 2023-03-08 19:00:22,884 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN 2023-03-08 19:00:22,884 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN 2023-03-08 19:00:22,885 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN 2023-03-08 19:00:22,886 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN 2023-03-08 19:00:22,887 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:22,887 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:22,887 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,33875,1678302017939; forceNewPlan=false, retain=true 2023-03-08 19:00:22,887 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,46129,1678302018069; forceNewPlan=false, retain=true 2023-03-08 19:00:22,887 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN 2023-03-08 19:00:22,887 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:22,887 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:22,887 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,46129,1678302018069; forceNewPlan=false, retain=true 2023-03-08 19:00:22,888 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,41161,1678302018011; forceNewPlan=false, retain=true 2023-03-08 19:00:22,888 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN 2023-03-08 19:00:22,888 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:22,888 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,41161,1678302018011; forceNewPlan=false, retain=true 2023-03-08 19:00:22,890 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:22,890 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,33875,1678302017939; forceNewPlan=false, retain=true 2023-03-08 19:00:23,038 INFO [jenkins-hbase6:43879] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-03-08 19:00:23,038 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:23,038 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:23,038 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:23,038 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:23,038 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:23,039 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302023038"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302023038"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302023038"}]},"ts":"1678302023038"} 2023-03-08 19:00:23,039 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302023038"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302023038"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302023038"}]},"ts":"1678302023038"} 2023-03-08 19:00:23,039 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302023038"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302023038"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302023038"}]},"ts":"1678302023038"} 2023-03-08 19:00:23,038 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302023038"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302023038"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302023038"}]},"ts":"1678302023038"} 2023-03-08 19:00:23,039 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302023038"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302023038"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302023038"}]},"ts":"1678302023038"} 2023-03-08 19:00:23,041 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=40, state=RUNNABLE; OpenRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:23,043 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=37, state=RUNNABLE; OpenRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:23,044 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=38, state=RUNNABLE; OpenRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:23,045 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=42, state=RUNNABLE; OpenRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:23,047 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:23,047 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302023047"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302023047"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302023047"}]},"ts":"1678302023047"} 2023-03-08 19:00:23,048 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=39, state=RUNNABLE; OpenRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:23,051 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=41, state=RUNNABLE; OpenRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:23,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-03-08 19:00:23,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:23,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'} 2023-03-08 19:00:23,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:23,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:23,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:23,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:23,201 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:23,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'} 2023-03-08 19:00:23,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:23,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:23,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:23,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:23,203 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:23,205 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf 2023-03-08 19:00:23,205 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf 2023-03-08 19:00:23,205 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b8add49535b4b0366d051f3a48b2a333 columnFamilyName cf 2023-03-08 19:00:23,207 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:23,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''} 2023-03-08 19:00:23,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:23,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:23,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:23,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:23,210 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(310): Store=b8add49535b4b0366d051f3a48b2a333/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:23,216 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:23,216 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:23,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:23,218 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf 2023-03-08 19:00:23,218 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf 2023-03-08 19:00:23,219 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf 2023-03-08 19:00:23,219 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf 2023-03-08 19:00:23,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:23,220 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 668bd868cc8589cf26791b7dda1212d8 columnFamilyName cf 2023-03-08 19:00:23,220 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e3eaed3ac05b4e65948c1d4b5c75462d columnFamilyName cf 2023-03-08 19:00:23,222 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(310): Store=668bd868cc8589cf26791b7dda1212d8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:23,226 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(310): Store=e3eaed3ac05b4e65948c1d4b5c75462d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:23,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:23,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:23,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:23,231 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened b8add49535b4b0366d051f3a48b2a333; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72412011, jitterRate=0.07902304828166962}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:23,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:23,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for b8add49535b4b0366d051f3a48b2a333: 2023-03-08 19:00:23,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:23,233 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333., pid=43, masterSystemTime=1678302023194 2023-03-08 19:00:23,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:23,237 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:23,237 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:23,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'} 2023-03-08 19:00:23,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:23,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:23,238 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:23,238 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302023237"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302023237"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302023237"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302023237"}]},"ts":"1678302023237"} 2023-03-08 19:00:23,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:23,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:23,242 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:23,247 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN in 364 msec 2023-03-08 19:00:23,244 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=40 2023-03-08 19:00:23,252 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=40, state=SUCCESS; OpenRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,46129,1678302018069 in 200 msec 2023-03-08 19:00:23,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:23,251 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:23,254 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 668bd868cc8589cf26791b7dda1212d8; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59960029, jitterRate=-0.10652594268321991}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:23,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 668bd868cc8589cf26791b7dda1212d8: 2023-03-08 19:00:23,254 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened e3eaed3ac05b4e65948c1d4b5c75462d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61923644, jitterRate=-0.07726579904556274}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:23,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for e3eaed3ac05b4e65948c1d4b5c75462d: 2023-03-08 19:00:23,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d., pid=46, masterSystemTime=1678302023202 2023-03-08 19:00:23,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8., pid=44, masterSystemTime=1678302023196 2023-03-08 19:00:23,257 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf 2023-03-08 19:00:23,257 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf 2023-03-08 19:00:23,257 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region be16768f5782c50d4c94d5fdc74d3f28 columnFamilyName cf 2023-03-08 19:00:23,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:23,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:23,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:23,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'} 2023-03-08 19:00:23,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:23,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:23,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:23,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:23,260 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(310): Store=be16768f5782c50d4c94d5fdc74d3f28/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:23,262 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:23,262 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302023262"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302023262"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302023262"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302023262"}]},"ts":"1678302023262"} 2023-03-08 19:00:23,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:23,264 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:23,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:23,268 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:23,268 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302023268"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302023268"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302023268"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302023268"}]},"ts":"1678302023268"} 2023-03-08 19:00:23,269 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:23,270 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:23,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'} 2023-03-08 19:00:23,271 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf 2023-03-08 19:00:23,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:23,271 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf 2023-03-08 19:00:23,272 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7cdf03195577655a4e5d1e7f4ff92392 columnFamilyName cf 2023-03-08 19:00:23,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:23,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:23,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:23,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:23,274 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(310): Store=7cdf03195577655a4e5d1e7f4ff92392/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:23,278 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:23,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:23,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:23,278 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=42 2023-03-08 19:00:23,279 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=42, state=SUCCESS; OpenRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 229 msec 2023-03-08 19:00:23,280 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf 2023-03-08 19:00:23,280 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened be16768f5782c50d4c94d5fdc74d3f28; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70165810, jitterRate=0.04555204510688782}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:23,280 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf 2023-03-08 19:00:23,280 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=37 2023-03-08 19:00:23,280 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=37, state=SUCCESS; OpenRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939 in 233 msec 2023-03-08 19:00:23,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:23,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for be16768f5782c50d4c94d5fdc74d3f28: 2023-03-08 19:00:23,281 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ea43d1afb394fd7a05814d2275bdeffd columnFamilyName cf 2023-03-08 19:00:23,282 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN in 399 msec 2023-03-08 19:00:23,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28., pid=45, masterSystemTime=1678302023194 2023-03-08 19:00:23,282 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(310): Store=ea43d1afb394fd7a05814d2275bdeffd/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:23,284 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN in 400 msec 2023-03-08 19:00:23,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:23,284 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:23,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:23,285 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:23,285 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302023285"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302023285"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302023285"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302023285"}]},"ts":"1678302023285"} 2023-03-08 19:00:23,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:23,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:23,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 7cdf03195577655a4e5d1e7f4ff92392; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66854716, jitterRate=-0.003787100315093994}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:23,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 7cdf03195577655a4e5d1e7f4ff92392: 2023-03-08 19:00:23,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392., pid=47, masterSystemTime=1678302023202 2023-03-08 19:00:23,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:23,291 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:23,292 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=38 2023-03-08 19:00:23,293 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=38, state=SUCCESS; OpenRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,46129,1678302018069 in 244 msec 2023-03-08 19:00:23,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:23,293 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:23,293 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302023293"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302023293"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302023293"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302023293"}]},"ts":"1678302023293"} 2023-03-08 19:00:23,294 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened ea43d1afb394fd7a05814d2275bdeffd; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59616829, jitterRate=-0.11164002120494843}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:23,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for ea43d1afb394fd7a05814d2275bdeffd: 2023-03-08 19:00:23,295 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN in 411 msec 2023-03-08 19:00:23,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd., pid=48, masterSystemTime=1678302023196 2023-03-08 19:00:23,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:23,298 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:23,299 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=39 2023-03-08 19:00:23,299 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=39, state=SUCCESS; OpenRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,41161,1678302018011 in 248 msec 2023-03-08 19:00:23,299 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:23,299 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302023299"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302023299"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302023299"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302023299"}]},"ts":"1678302023299"} 2023-03-08 19:00:23,301 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN in 419 msec 2023-03-08 19:00:23,304 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=41 2023-03-08 19:00:23,304 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=41, state=SUCCESS; OpenRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,33875,1678302017939 in 250 msec 2023-03-08 19:00:23,306 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=36 2023-03-08 19:00:23,306 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN in 424 msec 2023-03-08 19:00:23,307 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302023307"}]},"ts":"1678302023307"} 2023-03-08 19:00:23,309 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=ENABLED in hbase:meta 2023-03-08 19:00:23,311 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1678302020890' was successfully enabled. 2023-03-08 19:00:23,316 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=36, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1678302020890 in 450 msec 2023-03-08 19:00:23,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-03-08 19:00:23,628 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1678302020890, procId: 36 completed 2023-03-08 19:00:23,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HRegion(8805): writing data to region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:23,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HRegion(8805): writing data to region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:23,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HRegion(8805): writing data to region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:23,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HRegion(8805): writing data to region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:23,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HRegion(8805): writing data to region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:23,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HRegion(8805): writing data to region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:23,692 DEBUG [Listener at localhost/45021] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1678302020890 2023-03-08 19:00:23,692 DEBUG [Listener at localhost/45021] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:23,694 INFO [Listener at localhost/45021] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-03-08 19:00:23,722 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-03-08 19:00:23,725 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-03-08 19:00:23,728 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-03-08 19:00:23,731 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-03-08 19:00:23,733 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-03-08 19:00:23,748 INFO [Listener at localhost/45021] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1678302020890 2023-03-08 19:00:23,748 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$11(2418): Client=jenkins//172.31.2.20 disable testRestoreSnapshot-1678302020890 2023-03-08 19:00:23,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:23,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-03-08 19:00:23,756 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302023756"}]},"ts":"1678302023756"} 2023-03-08 19:00:23,760 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=DISABLING in hbase:meta 2023-03-08 19:00:23,763 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1678302020890 to state=DISABLING 2023-03-08 19:00:23,765 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN}, {pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN}, {pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN}, {pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN}, {pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN}, {pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN}] 2023-03-08 19:00:23,768 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN 2023-03-08 19:00:23,768 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN 2023-03-08 19:00:23,768 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN 2023-03-08 19:00:23,768 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN 2023-03-08 19:00:23,769 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN 2023-03-08 19:00:23,770 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:23,770 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:23,770 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302023770"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302023770"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302023770"}]},"ts":"1678302023770"} 2023-03-08 19:00:23,770 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302023770"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302023770"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302023770"}]},"ts":"1678302023770"} 2023-03-08 19:00:23,770 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:23,770 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:23,771 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302023770"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302023770"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302023770"}]},"ts":"1678302023770"} 2023-03-08 19:00:23,771 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302023770"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302023770"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302023770"}]},"ts":"1678302023770"} 2023-03-08 19:00:23,772 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:23,772 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302023772"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302023772"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302023772"}]},"ts":"1678302023772"} 2023-03-08 19:00:23,773 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=53, state=RUNNABLE; CloseRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:23,775 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=51, state=RUNNABLE; CloseRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:23,777 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=50, state=RUNNABLE; CloseRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:23,778 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=52, state=RUNNABLE; CloseRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:23,778 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN 2023-03-08 19:00:23,779 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:23,779 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302023779"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302023779"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302023779"}]},"ts":"1678302023779"} 2023-03-08 19:00:23,780 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=55, state=RUNNABLE; CloseRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:23,782 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=54, state=RUNNABLE; CloseRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:23,927 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:23,928 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing be16768f5782c50d4c94d5fdc74d3f28, disabling compactions & flushes 2023-03-08 19:00:23,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:23,928 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:23,928 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. after waiting 0 ms 2023-03-08 19:00:23,928 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:23,929 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing be16768f5782c50d4c94d5fdc74d3f28 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-03-08 19:00:23,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:23,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing ea43d1afb394fd7a05814d2275bdeffd, disabling compactions & flushes 2023-03-08 19:00:23,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:23,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:23,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. after waiting 0 ms 2023-03-08 19:00:23,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:23,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing ea43d1afb394fd7a05814d2275bdeffd 1/1 column families, dataSize=2.81 KB heapSize=6.30 KB 2023-03-08 19:00:23,932 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:23,933 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 7cdf03195577655a4e5d1e7f4ff92392, disabling compactions & flushes 2023-03-08 19:00:23,933 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:23,933 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:23,933 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. after waiting 0 ms 2023-03-08 19:00:23,933 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:23,933 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 7cdf03195577655a4e5d1e7f4ff92392 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-03-08 19:00:24,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-03-08 19:00:24,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 2023-03-08 19:00:24,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 2023-03-08 19:00:24,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc 2023-03-08 19:00:24,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/.tmp/cf/7e72d8ad17d24d9f8bfdc8e8a32a99ed 2023-03-08 19:00:24,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/.tmp/cf/57dba886df0248f48f93263d56e12c12 2023-03-08 19:00:24,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/.tmp/cf/f24cdc04e3d949cf8fcfbd4389e9cdfb 2023-03-08 19:00:24,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/.tmp/cf/f24cdc04e3d949cf8fcfbd4389e9cdfb as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/f24cdc04e3d949cf8fcfbd4389e9cdfb 2023-03-08 19:00:24,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/.tmp/cf/57dba886df0248f48f93263d56e12c12 as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/57dba886df0248f48f93263d56e12c12 2023-03-08 19:00:24,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/.tmp/cf/7e72d8ad17d24d9f8bfdc8e8a32a99ed as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/7e72d8ad17d24d9f8bfdc8e8a32a99ed 2023-03-08 19:00:24,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/f24cdc04e3d949cf8fcfbd4389e9cdfb, entries=32, sequenceid=8, filesize=10.4 K 2023-03-08 19:00:24,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/7e72d8ad17d24d9f8bfdc8e8a32a99ed, entries=43, sequenceid=8, filesize=12.2 K 2023-03-08 19:00:24,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/57dba886df0248f48f93263d56e12c12, entries=39, sequenceid=8, filesize=11.5 K 2023-03-08 19:00:24,127 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 7cdf03195577655a4e5d1e7f4ff92392 in 194ms, sequenceid=8, compaction requested=false 2023-03-08 19:00:24,127 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.81 KB/2879, heapSize ~6.28 KB/6432, currentSize=0 B/0 for ea43d1afb394fd7a05814d2275bdeffd in 197ms, sequenceid=8, compaction requested=false 2023-03-08 19:00:24,127 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for be16768f5782c50d4c94d5fdc74d3f28 in 198ms, sequenceid=8, compaction requested=false 2023-03-08 19:00:24,129 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1678302020890' 2023-03-08 19:00:24,129 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1678302020890' 2023-03-08 19:00:24,129 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1678302020890' 2023-03-08 19:00:24,143 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-03-08 19:00:24,143 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-03-08 19:00:24,143 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-03-08 19:00:24,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:24,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 7cdf03195577655a4e5d1e7f4ff92392: 2023-03-08 19:00:24,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:24,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:24,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for be16768f5782c50d4c94d5fdc74d3f28: 2023-03-08 19:00:24,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for ea43d1afb394fd7a05814d2275bdeffd: 2023-03-08 19:00:24,146 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:24,146 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:24,147 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing e3eaed3ac05b4e65948c1d4b5c75462d, disabling compactions & flushes 2023-03-08 19:00:24,147 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:24,147 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:24,147 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. after waiting 0 ms 2023-03-08 19:00:24,147 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:24,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing e3eaed3ac05b4e65948c1d4b5c75462d 1/1 column families, dataSize=22.11 KB heapSize=47.78 KB 2023-03-08 19:00:24,148 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=CLOSED 2023-03-08 19:00:24,148 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302024148"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302024148"}]},"ts":"1678302024148"} 2023-03-08 19:00:24,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:24,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:24,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 668bd868cc8589cf26791b7dda1212d8, disabling compactions & flushes 2023-03-08 19:00:24,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:24,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:24,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. after waiting 0 ms 2023-03-08 19:00:24,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:24,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 668bd868cc8589cf26791b7dda1212d8 1/1 column families, dataSize=1.37 KB heapSize=3.20 KB 2023-03-08 19:00:24,150 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=CLOSED 2023-03-08 19:00:24,150 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302024150"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302024150"}]},"ts":"1678302024150"} 2023-03-08 19:00:24,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:24,151 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:24,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing b8add49535b4b0366d051f3a48b2a333, disabling compactions & flushes 2023-03-08 19:00:24,152 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:24,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:24,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. after waiting 0 ms 2023-03-08 19:00:24,152 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=CLOSED 2023-03-08 19:00:24,152 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302024152"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302024152"}]},"ts":"1678302024152"} 2023-03-08 19:00:24,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:24,152 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing b8add49535b4b0366d051f3a48b2a333 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-03-08 19:00:24,155 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=52 2023-03-08 19:00:24,155 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=52, state=SUCCESS; CloseRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,41161,1678302018011 in 372 msec 2023-03-08 19:00:24,162 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN in 390 msec 2023-03-08 19:00:24,163 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=54 2023-03-08 19:00:24,164 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=54, state=SUCCESS; CloseRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,33875,1678302017939 in 372 msec 2023-03-08 19:00:24,165 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=51 2023-03-08 19:00:24,165 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=51, state=SUCCESS; CloseRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,46129,1678302018069 in 381 msec 2023-03-08 19:00:24,166 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN in 398 msec 2023-03-08 19:00:24,167 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN in 400 msec 2023-03-08 19:00:24,189 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 2023-03-08 19:00:24,200 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf 2023-03-08 19:00:24,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 2023-03-08 19:00:24,212 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/.tmp/cf/2a0e9bd776cc4f468bd20a74e4f61006 2023-03-08 19:00:24,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/.tmp/cf/9549786755914507847487aebb599d51 2023-03-08 19:00:24,225 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/.tmp/cf/2a0e9bd776cc4f468bd20a74e4f61006 as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/2a0e9bd776cc4f468bd20a74e4f61006 2023-03-08 19:00:24,228 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/.tmp/cf/9549786755914507847487aebb599d51 as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/9549786755914507847487aebb599d51 2023-03-08 19:00:24,230 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/.tmp/cf/bc3874f6411e445bb582265e908e860e 2023-03-08 19:00:24,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/2a0e9bd776cc4f468bd20a74e4f61006, entries=21, sequenceid=8, filesize=8.5 K 2023-03-08 19:00:24,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.37 KB/1405, heapSize ~3.19 KB/3264, currentSize=0 B/0 for 668bd868cc8589cf26791b7dda1212d8 in 91ms, sequenceid=8, compaction requested=false 2023-03-08 19:00:24,245 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/9549786755914507847487aebb599d51, entries=27, sequenceid=8, filesize=9.5 K 2023-03-08 19:00:24,246 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/.tmp/cf/bc3874f6411e445bb582265e908e860e as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/bc3874f6411e445bb582265e908e860e 2023-03-08 19:00:24,246 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for b8add49535b4b0366d051f3a48b2a333 in 94ms, sequenceid=8, compaction requested=false 2023-03-08 19:00:24,254 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-03-08 19:00:24,255 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:24,255 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 668bd868cc8589cf26791b7dda1212d8: 2023-03-08 19:00:24,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:24,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/bc3874f6411e445bb582265e908e860e, entries=338, sequenceid=8, filesize=62.5 K 2023-03-08 19:00:24,260 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-03-08 19:00:24,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~22.11 KB/22642, heapSize ~47.77 KB/48912, currentSize=0 B/0 for e3eaed3ac05b4e65948c1d4b5c75462d in 113ms, sequenceid=8, compaction requested=false 2023-03-08 19:00:24,261 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=CLOSED 2023-03-08 19:00:24,261 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302024261"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302024261"}]},"ts":"1678302024261"} 2023-03-08 19:00:24,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:24,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for b8add49535b4b0366d051f3a48b2a333: 2023-03-08 19:00:24,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:24,268 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=CLOSED 2023-03-08 19:00:24,268 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302024268"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302024268"}]},"ts":"1678302024268"} 2023-03-08 19:00:24,268 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=50 2023-03-08 19:00:24,268 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=50, state=SUCCESS; CloseRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939 in 488 msec 2023-03-08 19:00:24,271 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN in 504 msec 2023-03-08 19:00:24,272 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=53 2023-03-08 19:00:24,273 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=53, state=SUCCESS; CloseRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,46129,1678302018069 in 497 msec 2023-03-08 19:00:24,277 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN in 508 msec 2023-03-08 19:00:24,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-03-08 19:00:24,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:24,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for e3eaed3ac05b4e65948c1d4b5c75462d: 2023-03-08 19:00:24,284 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:24,284 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=CLOSED 2023-03-08 19:00:24,285 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302024284"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302024284"}]},"ts":"1678302024284"} 2023-03-08 19:00:24,289 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=55 2023-03-08 19:00:24,289 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=55, state=SUCCESS; CloseRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 506 msec 2023-03-08 19:00:24,291 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=49 2023-03-08 19:00:24,291 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN in 524 msec 2023-03-08 19:00:24,292 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302024292"}]},"ts":"1678302024292"} 2023-03-08 19:00:24,294 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=DISABLED in hbase:meta 2023-03-08 19:00:24,296 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1678302020890 to state=DISABLED 2023-03-08 19:00:24,299 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1678302020890 in 548 msec 2023-03-08 19:00:24,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-03-08 19:00:24,510 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1678302020890, procId: 49 completed 2023-03-08 19:00:24,511 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1580): Client=jenkins//172.31.2.20 snapshot request for:{ ss=snaptb0-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:24,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1678302024511 (current time:1678302024511). 2023-03-08 19:00:24,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-03-08 19:00:24,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] zookeeper.ReadOnlyZKClient(139): Connect 0x2edcde19 to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:24,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@794413c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:24,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:24,521 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:42104, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:24,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2edcde19 to 127.0.0.1:59169 2023-03-08 19:00:24,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:24,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-03-08 19:00:24,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:24,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:24,526 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:24,527 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE; timeout=600000, timestamp=1678302624527 2023-03-08 19:00:24,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:24,527 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1678302020890 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1678302020890 2023-03-08 19:00:24,527 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-03-08 19:00:24,528 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-03-08 19:00:24,544 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1678302020890 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } 2023-03-08 19:00:24,546 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:24,546 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:24,546 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:24,546 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:24,546 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:24,546 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:24,546 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:24,546 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:24,546 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:24,547 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:24,547 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:24,546 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:24,547 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:24,547 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:24,549 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/7e72d8ad17d24d9f8bfdc8e8a32a99ed] hfiles 2023-03-08 19:00:24,549 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/7e72d8ad17d24d9f8bfdc8e8a32a99ed 2023-03-08 19:00:24,549 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/2a0e9bd776cc4f468bd20a74e4f61006] hfiles 2023-03-08 19:00:24,549 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/2a0e9bd776cc4f468bd20a74e4f61006 2023-03-08 19:00:24,550 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/9549786755914507847487aebb599d51] hfiles 2023-03-08 19:00:24,550 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/9549786755914507847487aebb599d51 2023-03-08 19:00:24,550 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/57dba886df0248f48f93263d56e12c12] hfiles 2023-03-08 19:00:24,550 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/57dba886df0248f48f93263d56e12c12 2023-03-08 19:00:24,551 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/f24cdc04e3d949cf8fcfbd4389e9cdfb] hfiles 2023-03-08 19:00:24,551 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/f24cdc04e3d949cf8fcfbd4389e9cdfb 2023-03-08 19:00:24,551 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/bc3874f6411e445bb582265e908e860e] hfiles 2023-03-08 19:00:24,551 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/bc3874f6411e445bb582265e908e860e 2023-03-08 19:00:24,553 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf] hfiles 2023-03-08 19:00:24,553 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 2023-03-08 19:00:24,553 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 2023-03-08 19:00:24,553 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc 2023-03-08 19:00:24,553 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 2023-03-08 19:00:24,553 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 2023-03-08 19:00:24,553 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf 2023-03-08 19:00:24,591 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1678302020890 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } as finished. 2023-03-08 19:00:24,592 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1678302020890 2023-03-08 19:00:24,593 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/snaptb0-1678302020890 2023-03-08 19:00:24,651 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/snaptb0-1678302020890 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/snaptb0-1678302020890 2023-03-08 19:00:24,677 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1678302020890 of table testRestoreSnapshot-1678302020890 completed 2023-03-08 19:00:24,677 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/snaptb0-1678302020890 2023-03-08 19:00:24,677 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/snaptb0-1678302020890 2023-03-08 19:00:24,677 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1678302020890 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1678302020890 at 1678302024527Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1678302020890 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } at 1678302024544 (+17 ms)Consolidate snapshot: snaptb0-1678302020890 at 1678302024592 (+48 ms)Loading Region manifests for snaptb0-1678302020890 at 1678302024592Writing data manifest for snaptb0-1678302020890 at 1678302024607 (+15 ms)Verifying snapshot: snaptb0-1678302020890 at 1678302024629 (+22 ms)Snapshot snaptb0-1678302020890 of table testRestoreSnapshot-1678302020890 completed at 1678302024677 (+48 ms) 2023-03-08 19:00:24,679 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:24,681 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE in 155 msec 2023-03-08 19:00:24,778 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-03-08 19:00:24,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } is done 2023-03-08 19:00:24,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 }' has completed, notifying client. 2023-03-08 19:00:24,780 INFO [Listener at localhost/45021] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1678302020890 2023-03-08 19:00:24,781 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$10(2385): Client=jenkins//172.31.2.20 enable testRestoreSnapshot-1678302020890 2023-03-08 19:00:24,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:24,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-03-08 19:00:24,787 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1678302020890 2023-03-08 19:00:24,787 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302024787"}]},"ts":"1678302024787"} 2023-03-08 19:00:24,789 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=ENABLING in hbase:meta 2023-03-08 19:00:24,796 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-03-08 19:00:24,796 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN}, {pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN}, {pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN}, {pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN}, {pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN}] 2023-03-08 19:00:24,799 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN 2023-03-08 19:00:24,799 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN 2023-03-08 19:00:24,799 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN 2023-03-08 19:00:24,799 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN 2023-03-08 19:00:24,800 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN 2023-03-08 19:00:24,800 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:24,800 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:24,800 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,33875,1678302017939; forceNewPlan=false, retain=true 2023-03-08 19:00:24,800 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:24,800 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,41161,1678302018011; forceNewPlan=false, retain=true 2023-03-08 19:00:24,801 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:24,801 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,41161,1678302018011; forceNewPlan=false, retain=true 2023-03-08 19:00:24,800 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,46129,1678302018069; forceNewPlan=false, retain=true 2023-03-08 19:00:24,801 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:24,801 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,46129,1678302018069; forceNewPlan=false, retain=true 2023-03-08 19:00:24,801 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN 2023-03-08 19:00:24,803 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:24,803 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,33875,1678302017939; forceNewPlan=false, retain=true 2023-03-08 19:00:24,951 INFO [jenkins-hbase6:43879] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-03-08 19:00:24,951 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:24,951 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:24,951 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302024951"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302024951"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302024951"}]},"ts":"1678302024951"} 2023-03-08 19:00:24,951 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:24,952 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302024951"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302024951"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302024951"}]},"ts":"1678302024951"} 2023-03-08 19:00:24,951 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:24,951 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:24,952 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302024951"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302024951"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302024951"}]},"ts":"1678302024951"} 2023-03-08 19:00:24,952 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302024951"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302024951"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302024951"}]},"ts":"1678302024951"} 2023-03-08 19:00:24,952 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302024951"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302024951"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302024951"}]},"ts":"1678302024951"} 2023-03-08 19:00:24,954 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=65, state=RUNNABLE; OpenRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:24,955 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=64, state=RUNNABLE; OpenRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:24,956 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=66, state=RUNNABLE; OpenRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:24,957 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=69, state=RUNNABLE; OpenRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:24,958 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=67, state=RUNNABLE; OpenRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:24,960 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:24,960 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302024960"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302024960"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302024960"}]},"ts":"1678302024960"} 2023-03-08 19:00:24,965 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=68, state=RUNNABLE; OpenRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:25,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-03-08 19:00:25,115 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:25,115 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'} 2023-03-08 19:00:25,115 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:25,115 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:25,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:25,116 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:25,116 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:25,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'} 2023-03-08 19:00:25,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'} 2023-03-08 19:00:25,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:25,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:25,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:25,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:25,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:25,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:25,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:25,118 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:25,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:25,120 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:25,122 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:25,122 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:25,122 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf 2023-03-08 19:00:25,122 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf 2023-03-08 19:00:25,123 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7cdf03195577655a4e5d1e7f4ff92392 columnFamilyName cf 2023-03-08 19:00:25,129 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf 2023-03-08 19:00:25,129 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf 2023-03-08 19:00:25,129 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf 2023-03-08 19:00:25,130 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf 2023-03-08 19:00:25,130 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 668bd868cc8589cf26791b7dda1212d8 columnFamilyName cf 2023-03-08 19:00:25,130 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b8add49535b4b0366d051f3a48b2a333 columnFamilyName cf 2023-03-08 19:00:25,135 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/f24cdc04e3d949cf8fcfbd4389e9cdfb 2023-03-08 19:00:25,137 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(310): Store=7cdf03195577655a4e5d1e7f4ff92392/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:25,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:25,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:25,143 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/2a0e9bd776cc4f468bd20a74e4f61006 2023-03-08 19:00:25,143 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(310): Store=668bd868cc8589cf26791b7dda1212d8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:25,147 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/9549786755914507847487aebb599d51 2023-03-08 19:00:25,147 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(310): Store=b8add49535b4b0366d051f3a48b2a333/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:25,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:25,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:25,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:25,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:25,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:25,153 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 7cdf03195577655a4e5d1e7f4ff92392; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60054645, jitterRate=-0.10511605441570282}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:25,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 7cdf03195577655a4e5d1e7f4ff92392: 2023-03-08 19:00:25,155 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392., pid=72, masterSystemTime=1678302025110 2023-03-08 19:00:25,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:25,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:25,157 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 668bd868cc8589cf26791b7dda1212d8; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61303706, jitterRate=-0.08650359511375427}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:25,157 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 668bd868cc8589cf26791b7dda1212d8: 2023-03-08 19:00:25,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:25,158 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened b8add49535b4b0366d051f3a48b2a333; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68574650, jitterRate=0.02184191346168518}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:25,158 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:25,158 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8., pid=71, masterSystemTime=1678302025108 2023-03-08 19:00:25,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for b8add49535b4b0366d051f3a48b2a333: 2023-03-08 19:00:25,159 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:25,159 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:25,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''} 2023-03-08 19:00:25,159 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302025159"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302025159"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302025159"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302025159"}]},"ts":"1678302025159"} 2023-03-08 19:00:25,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:25,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:25,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:25,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:25,161 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333., pid=74, masterSystemTime=1678302025107 2023-03-08 19:00:25,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:25,162 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:25,162 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:25,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'} 2023-03-08 19:00:25,162 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:25,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:25,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:25,162 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302025162"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302025162"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302025162"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302025162"}]},"ts":"1678302025162"} 2023-03-08 19:00:25,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:25,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:25,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:25,165 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:25,166 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:25,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'} 2023-03-08 19:00:25,166 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:25,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:25,166 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302025166"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302025166"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302025166"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302025166"}]},"ts":"1678302025166"} 2023-03-08 19:00:25,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:25,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:25,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:25,167 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=66 2023-03-08 19:00:25,167 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=66, state=SUCCESS; OpenRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,41161,1678302018011 in 207 msec 2023-03-08 19:00:25,168 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:25,169 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=64 2023-03-08 19:00:25,169 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=64, state=SUCCESS; OpenRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939 in 210 msec 2023-03-08 19:00:25,170 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN in 371 msec 2023-03-08 19:00:25,171 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN in 373 msec 2023-03-08 19:00:25,171 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=67 2023-03-08 19:00:25,171 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=67, state=SUCCESS; OpenRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,46129,1678302018069 in 210 msec 2023-03-08 19:00:25,173 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN in 375 msec 2023-03-08 19:00:25,174 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:25,174 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:25,175 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf 2023-03-08 19:00:25,175 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf 2023-03-08 19:00:25,176 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e3eaed3ac05b4e65948c1d4b5c75462d columnFamilyName cf 2023-03-08 19:00:25,176 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf 2023-03-08 19:00:25,176 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf 2023-03-08 19:00:25,176 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf 2023-03-08 19:00:25,176 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf 2023-03-08 19:00:25,176 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ea43d1afb394fd7a05814d2275bdeffd columnFamilyName cf 2023-03-08 19:00:25,177 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region be16768f5782c50d4c94d5fdc74d3f28 columnFamilyName cf 2023-03-08 19:00:25,184 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/bc3874f6411e445bb582265e908e860e 2023-03-08 19:00:25,184 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(310): Store=e3eaed3ac05b4e65948c1d4b5c75462d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:25,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:25,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:25,189 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/7e72d8ad17d24d9f8bfdc8e8a32a99ed 2023-03-08 19:00:25,190 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(310): Store=ea43d1afb394fd7a05814d2275bdeffd/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:25,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:25,191 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/57dba886df0248f48f93263d56e12c12 2023-03-08 19:00:25,192 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(310): Store=be16768f5782c50d4c94d5fdc74d3f28/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:25,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:25,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:25,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:25,194 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened e3eaed3ac05b4e65948c1d4b5c75462d; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66416100, jitterRate=-0.010322988033294678}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:25,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for e3eaed3ac05b4e65948c1d4b5c75462d: 2023-03-08 19:00:25,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:25,195 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d., pid=73, masterSystemTime=1678302025110 2023-03-08 19:00:25,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:25,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:25,199 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:25,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened ea43d1afb394fd7a05814d2275bdeffd; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71242424, jitterRate=0.06159484386444092}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:25,200 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:25,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for ea43d1afb394fd7a05814d2275bdeffd: 2023-03-08 19:00:25,200 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302025200"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302025200"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302025200"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302025200"}]},"ts":"1678302025200"} 2023-03-08 19:00:25,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:25,201 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd., pid=75, masterSystemTime=1678302025108 2023-03-08 19:00:25,202 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened be16768f5782c50d4c94d5fdc74d3f28; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60354273, jitterRate=-0.10065124928951263}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:25,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for be16768f5782c50d4c94d5fdc74d3f28: 2023-03-08 19:00:25,203 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28., pid=70, masterSystemTime=1678302025107 2023-03-08 19:00:25,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:25,204 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:25,204 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:25,205 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302025204"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302025204"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302025204"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302025204"}]},"ts":"1678302025204"} 2023-03-08 19:00:25,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:25,206 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:25,206 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=69 2023-03-08 19:00:25,206 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:25,206 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=69, state=SUCCESS; OpenRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 246 msec 2023-03-08 19:00:25,207 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302025206"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302025206"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302025206"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302025206"}]},"ts":"1678302025206"} 2023-03-08 19:00:25,209 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=69, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN in 411 msec 2023-03-08 19:00:25,211 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=68 2023-03-08 19:00:25,211 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=68, state=SUCCESS; OpenRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,33875,1678302017939 in 244 msec 2023-03-08 19:00:25,213 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=65 2023-03-08 19:00:25,213 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=68, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN in 415 msec 2023-03-08 19:00:25,213 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=65, state=SUCCESS; OpenRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,46129,1678302018069 in 255 msec 2023-03-08 19:00:25,215 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=65, resume processing ppid=63 2023-03-08 19:00:25,215 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN in 417 msec 2023-03-08 19:00:25,216 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302025216"}]},"ts":"1678302025216"} 2023-03-08 19:00:25,218 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=ENABLED in hbase:meta 2023-03-08 19:00:25,220 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1678302020890' was successfully enabled. 2023-03-08 19:00:25,222 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1678302020890 in 440 msec 2023-03-08 19:00:25,372 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-08 19:00:25,429 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-08 19:00:25,430 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-08 19:00:25,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-03-08 19:00:25,539 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1678302020890, procId: 63 completed 2023-03-08 19:00:25,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HRegion(8805): writing data to region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:25,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HRegion(8805): writing data to region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:25,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HRegion(8805): writing data to region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:25,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HRegion(8805): writing data to region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:25,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HRegion(8805): writing data to region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:25,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HRegion(8805): writing data to region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:25,566 DEBUG [Listener at localhost/45021] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1678302020890 2023-03-08 19:00:25,566 DEBUG [Listener at localhost/45021] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:25,568 INFO [Listener at localhost/45021] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-03-08 19:00:25,757 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-03-08 19:00:25,957 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-03-08 19:00:26,119 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-03-08 19:00:26,241 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-03-08 19:00:26,458 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-03-08 19:00:26,764 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-08 19:00:26,765 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-08 19:00:27,983 DEBUG [Listener at localhost/45021] zookeeper.ReadOnlyZKClient(139): Connect 0x7b77b3f7 to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:27,990 DEBUG [Listener at localhost/45021] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20ceba7a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:27,992 DEBUG [hconnection-0x14908088-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:27,996 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:42112, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:28,077 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-03-08 19:00:28,080 DEBUG [hconnection-0x14908088-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:28,086 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:45414, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:28,223 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-03-08 19:00:28,228 DEBUG [hconnection-0x14908088-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:28,231 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:56348, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:28,359 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-03-08 19:00:28,479 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-03-08 19:00:28,631 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-03-08 19:00:29,905 INFO [Listener at localhost/45021] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1678302020890 2023-03-08 19:00:29,906 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$11(2418): Client=jenkins//172.31.2.20 disable testRestoreSnapshot-1678302020890 2023-03-08 19:00:29,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=76, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:29,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-03-08 19:00:29,911 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302029911"}]},"ts":"1678302029911"} 2023-03-08 19:00:29,914 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=DISABLING in hbase:meta 2023-03-08 19:00:29,916 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1678302020890 to state=DISABLING 2023-03-08 19:00:29,917 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN}, {pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN}, {pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN}, {pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN}, {pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN}, {pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN}] 2023-03-08 19:00:29,919 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN 2023-03-08 19:00:29,919 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN 2023-03-08 19:00:29,919 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN 2023-03-08 19:00:29,920 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:29,920 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN 2023-03-08 19:00:29,920 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN 2023-03-08 19:00:29,920 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:29,920 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302029920"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302029920"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302029920"}]},"ts":"1678302029920"} 2023-03-08 19:00:29,920 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302029920"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302029920"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302029920"}]},"ts":"1678302029920"} 2023-03-08 19:00:29,920 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:29,921 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302029920"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302029920"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302029920"}]},"ts":"1678302029920"} 2023-03-08 19:00:29,921 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:29,921 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:29,921 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302029921"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302029921"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302029921"}]},"ts":"1678302029921"} 2023-03-08 19:00:29,921 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302029921"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302029921"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302029921"}]},"ts":"1678302029921"} 2023-03-08 19:00:29,923 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=80, state=RUNNABLE; CloseRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:29,924 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=84, ppid=81, state=RUNNABLE; CloseRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:29,926 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=82, state=RUNNABLE; CloseRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:29,929 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=79, state=RUNNABLE; CloseRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:29,930 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN 2023-03-08 19:00:29,930 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=78, state=RUNNABLE; CloseRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:29,932 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:29,933 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302029932"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302029932"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302029932"}]},"ts":"1678302029932"} 2023-03-08 19:00:29,935 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=88, ppid=77, state=RUNNABLE; CloseRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:30,077 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:30,078 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing b8add49535b4b0366d051f3a48b2a333, disabling compactions & flushes 2023-03-08 19:00:30,078 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:30,078 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:30,078 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. after waiting 0 ms 2023-03-08 19:00:30,078 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:30,078 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing b8add49535b4b0366d051f3a48b2a333 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-03-08 19:00:30,080 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:30,080 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing ea43d1afb394fd7a05814d2275bdeffd, disabling compactions & flushes 2023-03-08 19:00:30,080 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:30,080 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:30,080 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. after waiting 0 ms 2023-03-08 19:00:30,080 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:30,080 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing ea43d1afb394fd7a05814d2275bdeffd 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-03-08 19:00:30,081 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:30,082 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing e3eaed3ac05b4e65948c1d4b5c75462d, disabling compactions & flushes 2023-03-08 19:00:30,082 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:30,082 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:30,082 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. after waiting 0 ms 2023-03-08 19:00:30,082 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:30,082 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing e3eaed3ac05b4e65948c1d4b5c75462d 1/1 column families, dataSize=21.13 KB heapSize=45.67 KB 2023-03-08 19:00:30,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 2023-03-08 19:00:30,107 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 2023-03-08 19:00:30,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 2023-03-08 19:00:30,113 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/.tmp/cf/172aadf8677145f0b28293dbaeda5b9f 2023-03-08 19:00:30,117 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/.tmp/cf/bd07ca0c4ad54ba0bb89f6a8507d4e9b 2023-03-08 19:00:30,119 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/.tmp/cf/172aadf8677145f0b28293dbaeda5b9f as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/172aadf8677145f0b28293dbaeda5b9f 2023-03-08 19:00:30,121 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/.tmp/cf/bd07ca0c4ad54ba0bb89f6a8507d4e9b as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/bd07ca0c4ad54ba0bb89f6a8507d4e9b 2023-03-08 19:00:30,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/172aadf8677145f0b28293dbaeda5b9f, entries=32, sequenceid=15, filesize=10.4 K 2023-03-08 19:00:30,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=21.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/.tmp/cf/64a5b55bb32e4bff96aba52c33665694 2023-03-08 19:00:30,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for b8add49535b4b0366d051f3a48b2a333 in 46ms, sequenceid=15, compaction requested=false 2023-03-08 19:00:30,127 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/bd07ca0c4ad54ba0bb89f6a8507d4e9b, entries=32, sequenceid=15, filesize=10.4 K 2023-03-08 19:00:30,128 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for ea43d1afb394fd7a05814d2275bdeffd in 48ms, sequenceid=15, compaction requested=false 2023-03-08 19:00:30,131 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/.tmp/cf/64a5b55bb32e4bff96aba52c33665694 as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/64a5b55bb32e4bff96aba52c33665694 2023-03-08 19:00:30,133 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-03-08 19:00:30,134 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:30,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for b8add49535b4b0366d051f3a48b2a333: 2023-03-08 19:00:30,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-03-08 19:00:30,135 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:30,136 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for ea43d1afb394fd7a05814d2275bdeffd: 2023-03-08 19:00:30,136 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/64a5b55bb32e4bff96aba52c33665694, entries=323, sequenceid=15, filesize=60.0 K 2023-03-08 19:00:30,137 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~21.13 KB/21637, heapSize ~45.66 KB/46752, currentSize=0 B/0 for e3eaed3ac05b4e65948c1d4b5c75462d in 55ms, sequenceid=15, compaction requested=false 2023-03-08 19:00:30,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:30,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:30,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing be16768f5782c50d4c94d5fdc74d3f28, disabling compactions & flushes 2023-03-08 19:00:30,142 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:30,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:30,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. after waiting 0 ms 2023-03-08 19:00:30,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:30,142 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing be16768f5782c50d4c94d5fdc74d3f28 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-03-08 19:00:30,142 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=CLOSED 2023-03-08 19:00:30,142 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302030142"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302030142"}]},"ts":"1678302030142"} 2023-03-08 19:00:30,143 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:30,143 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:30,143 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 668bd868cc8589cf26791b7dda1212d8, disabling compactions & flushes 2023-03-08 19:00:30,143 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:30,143 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=CLOSED 2023-03-08 19:00:30,143 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:30,143 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. after waiting 0 ms 2023-03-08 19:00:30,143 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:30,143 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302030143"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302030143"}]},"ts":"1678302030143"} 2023-03-08 19:00:30,143 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 668bd868cc8589cf26791b7dda1212d8 1/1 column families, dataSize=2.62 KB heapSize=5.88 KB 2023-03-08 19:00:30,147 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=80 2023-03-08 19:00:30,148 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=80, state=SUCCESS; CloseRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,46129,1678302018069 in 222 msec 2023-03-08 19:00:30,148 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=84, resume processing ppid=81 2023-03-08 19:00:30,148 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=84, ppid=81, state=SUCCESS; CloseRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,33875,1678302017939 in 222 msec 2023-03-08 19:00:30,149 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN in 232 msec 2023-03-08 19:00:30,150 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN in 233 msec 2023-03-08 19:00:30,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-03-08 19:00:30,162 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:30,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for e3eaed3ac05b4e65948c1d4b5c75462d: 2023-03-08 19:00:30,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-03-08 19:00:30,164 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:30,164 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:30,165 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 7cdf03195577655a4e5d1e7f4ff92392, disabling compactions & flushes 2023-03-08 19:00:30,165 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:30,165 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:30,165 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. after waiting 0 ms 2023-03-08 19:00:30,165 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:30,165 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 7cdf03195577655a4e5d1e7f4ff92392 1/1 column families, dataSize=2.62 KB heapSize=5.88 KB 2023-03-08 19:00:30,165 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=CLOSED 2023-03-08 19:00:30,165 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302030165"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302030165"}]},"ts":"1678302030165"} 2023-03-08 19:00:30,169 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=82 2023-03-08 19:00:30,169 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=82, state=SUCCESS; CloseRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 241 msec 2023-03-08 19:00:30,171 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN in 253 msec 2023-03-08 19:00:30,182 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d 2023-03-08 19:00:30,186 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 2023-03-08 19:00:30,189 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 2023-03-08 19:00:30,192 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/.tmp/cf/5b72a399cd494fa4836e4a3e20c6b1ba 2023-03-08 19:00:30,196 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/.tmp/cf/4943e5745d97426d8d62901400379e84 2023-03-08 19:00:30,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/.tmp/cf/5b72a399cd494fa4836e4a3e20c6b1ba as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/5b72a399cd494fa4836e4a3e20c6b1ba 2023-03-08 19:00:30,198 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/.tmp/cf/657092e48c0b4ad0ab0fa0681948dafb 2023-03-08 19:00:30,200 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/.tmp/cf/4943e5745d97426d8d62901400379e84 as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/4943e5745d97426d8d62901400379e84 2023-03-08 19:00:30,202 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/5b72a399cd494fa4836e4a3e20c6b1ba, entries=40, sequenceid=15, filesize=11.7 K 2023-03-08 19:00:30,202 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/.tmp/cf/657092e48c0b4ad0ab0fa0681948dafb as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/657092e48c0b4ad0ab0fa0681948dafb 2023-03-08 19:00:30,203 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.62 KB/2678, heapSize ~5.86 KB/6000, currentSize=0 B/0 for 668bd868cc8589cf26791b7dda1212d8 in 60ms, sequenceid=15, compaction requested=false 2023-03-08 19:00:30,207 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/4943e5745d97426d8d62901400379e84, entries=33, sequenceid=15, filesize=10.5 K 2023-03-08 19:00:30,207 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for be16768f5782c50d4c94d5fdc74d3f28 in 65ms, sequenceid=15, compaction requested=false 2023-03-08 19:00:30,212 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/657092e48c0b4ad0ab0fa0681948dafb, entries=40, sequenceid=15, filesize=11.7 K 2023-03-08 19:00:30,213 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.62 KB/2678, heapSize ~5.86 KB/6000, currentSize=0 B/0 for 7cdf03195577655a4e5d1e7f4ff92392 in 48ms, sequenceid=15, compaction requested=false 2023-03-08 19:00:30,214 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-03-08 19:00:30,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-03-08 19:00:30,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:30,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:30,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 668bd868cc8589cf26791b7dda1212d8: 2023-03-08 19:00:30,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for be16768f5782c50d4c94d5fdc74d3f28: 2023-03-08 19:00:30,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:30,221 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=CLOSED 2023-03-08 19:00:30,221 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302030221"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302030221"}]},"ts":"1678302030221"} 2023-03-08 19:00:30,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:30,222 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=CLOSED 2023-03-08 19:00:30,222 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302030222"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302030222"}]},"ts":"1678302030222"} 2023-03-08 19:00:30,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-03-08 19:00:30,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:30,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 7cdf03195577655a4e5d1e7f4ff92392: 2023-03-08 19:00:30,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:30,227 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=CLOSED 2023-03-08 19:00:30,227 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302030227"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302030227"}]},"ts":"1678302030227"} 2023-03-08 19:00:30,227 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=78 2023-03-08 19:00:30,228 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=78, state=SUCCESS; CloseRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,46129,1678302018069 in 293 msec 2023-03-08 19:00:30,228 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=88, resume processing ppid=77 2023-03-08 19:00:30,229 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=88, ppid=77, state=SUCCESS; CloseRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939 in 290 msec 2023-03-08 19:00:30,230 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN in 311 msec 2023-03-08 19:00:30,231 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN in 313 msec 2023-03-08 19:00:30,231 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=79 2023-03-08 19:00:30,231 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=79, state=SUCCESS; CloseRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,41161,1678302018011 in 300 msec 2023-03-08 19:00:30,233 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=79, resume processing ppid=76 2023-03-08 19:00:30,233 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=79, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN in 315 msec 2023-03-08 19:00:30,234 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302030234"}]},"ts":"1678302030234"} 2023-03-08 19:00:30,235 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=DISABLED in hbase:meta 2023-03-08 19:00:30,238 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1678302020890 to state=DISABLED 2023-03-08 19:00:30,240 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=76, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1678302020890 in 333 msec 2023-03-08 19:00:30,619 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-08 19:00:30,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-03-08 19:00:30,665 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1678302020890, procId: 76 completed 2023-03-08 19:00:30,666 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1580): Client=jenkins//172.31.2.20 snapshot request for:{ ss=snaptb1-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:30,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1678302030666 (current time:1678302030666). 2023-03-08 19:00:30,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-03-08 19:00:30,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] zookeeper.ReadOnlyZKClient(139): Connect 0x0fd7d3b2 to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:30,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@47b3e847, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:30,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:30,674 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:42114, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:30,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0fd7d3b2 to 127.0.0.1:59169 2023-03-08 19:00:30,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:30,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-03-08 19:00:30,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb1-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:30,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:30,680 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:30,681 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=89, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE; timeout=600000, timestamp=1678302630681 2023-03-08 19:00:30,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb1-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:30,681 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb1-1678302020890 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1678302020890 2023-03-08 19:00:30,681 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb1-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-03-08 19:00:30,682 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-03-08 19:00:30,697 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1678302020890 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } 2023-03-08 19:00:30,699 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:30,699 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:30,699 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:30,699 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:30,699 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:30,699 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:30,699 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:30,700 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:30,700 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:30,700 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:30,700 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:30,700 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:30,701 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:30,701 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:30,702 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/2a0e9bd776cc4f468bd20a74e4f61006, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/5b72a399cd494fa4836e4a3e20c6b1ba] hfiles 2023-03-08 19:00:30,702 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/2a0e9bd776cc4f468bd20a74e4f61006 2023-03-08 19:00:30,702 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/5b72a399cd494fa4836e4a3e20c6b1ba 2023-03-08 19:00:30,703 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/4943e5745d97426d8d62901400379e84, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/57dba886df0248f48f93263d56e12c12] hfiles 2023-03-08 19:00:30,703 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/4943e5745d97426d8d62901400379e84 2023-03-08 19:00:30,703 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/57dba886df0248f48f93263d56e12c12 2023-03-08 19:00:30,704 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/7e72d8ad17d24d9f8bfdc8e8a32a99ed, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/bd07ca0c4ad54ba0bb89f6a8507d4e9b] hfiles 2023-03-08 19:00:30,704 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/7e72d8ad17d24d9f8bfdc8e8a32a99ed 2023-03-08 19:00:30,704 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/bd07ca0c4ad54ba0bb89f6a8507d4e9b 2023-03-08 19:00:30,704 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/172aadf8677145f0b28293dbaeda5b9f, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/9549786755914507847487aebb599d51] hfiles 2023-03-08 19:00:30,704 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/172aadf8677145f0b28293dbaeda5b9f 2023-03-08 19:00:30,704 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/9549786755914507847487aebb599d51 2023-03-08 19:00:30,705 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/657092e48c0b4ad0ab0fa0681948dafb, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/f24cdc04e3d949cf8fcfbd4389e9cdfb] hfiles 2023-03-08 19:00:30,705 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/657092e48c0b4ad0ab0fa0681948dafb 2023-03-08 19:00:30,705 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/f24cdc04e3d949cf8fcfbd4389e9cdfb 2023-03-08 19:00:30,705 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/64a5b55bb32e4bff96aba52c33665694, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/bc3874f6411e445bb582265e908e860e] hfiles 2023-03-08 19:00:30,706 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/64a5b55bb32e4bff96aba52c33665694 2023-03-08 19:00:30,706 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/bc3874f6411e445bb582265e908e860e 2023-03-08 19:00:30,709 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf] hfiles 2023-03-08 19:00:30,709 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 2023-03-08 19:00:30,709 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 2023-03-08 19:00:30,709 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 2023-03-08 19:00:30,709 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 2023-03-08 19:00:30,709 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 2023-03-08 19:00:30,709 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc 2023-03-08 19:00:30,709 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d 2023-03-08 19:00:30,709 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 2023-03-08 19:00:30,709 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 2023-03-08 19:00:30,709 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 2023-03-08 19:00:30,709 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 2023-03-08 19:00:30,709 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf 2023-03-08 19:00:30,750 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb1-1678302020890 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } as finished. 2023-03-08 19:00:30,750 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb1-1678302020890 2023-03-08 19:00:30,751 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/snaptb1-1678302020890 2023-03-08 19:00:30,779 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/snaptb1-1678302020890 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/snaptb1-1678302020890 2023-03-08 19:00:30,808 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb1-1678302020890 of table testRestoreSnapshot-1678302020890 completed 2023-03-08 19:00:30,808 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/snaptb1-1678302020890 2023-03-08 19:00:30,808 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/snaptb1-1678302020890 2023-03-08 19:00:30,808 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb1-1678302020890 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1678302020890 at 1678302030681Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1678302020890 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } at 1678302030697 (+16 ms)Consolidate snapshot: snaptb1-1678302020890 at 1678302030750 (+53 ms)Loading Region manifests for snaptb1-1678302020890 at 1678302030751 (+1 ms)Writing data manifest for snaptb1-1678302020890 at 1678302030755 (+4 ms)Verifying snapshot: snaptb1-1678302020890 at 1678302030763 (+8 ms)Snapshot snaptb1-1678302020890 of table testRestoreSnapshot-1678302020890 completed at 1678302030808 (+45 ms) 2023-03-08 19:00:30,810 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=89, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:30,811 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=89, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE in 132 msec 2023-03-08 19:00:30,932 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-03-08 19:00:30,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } is done 2023-03-08 19:00:30,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb1-1678302020890 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 }' has completed, notifying client. 2023-03-08 19:00:30,950 INFO [Listener at localhost/45021] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb0-1678302020890-1678302030950 2023-03-08 19:00:30,951 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1580): Client=jenkins//172.31.2.20 snapshot request for:{ ss=hbase-failsafe-snaptb0-1678302020890-1678302030950 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:30,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1678302030951 (current time:1678302030951). 2023-03-08 19:00:30,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-03-08 19:00:30,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] zookeeper.ReadOnlyZKClient(139): Connect 0x0a995998 to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:30,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e2c1dc1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:30,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:30,960 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:42116, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:30,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a995998 to 127.0.0.1:59169 2023-03-08 19:00:30,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:30,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-03-08 19:00:30,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb0-1678302020890-1678302030950 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:30,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:30,964 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:30,965 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=90, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE; timeout=600000, timestamp=1678302630965 2023-03-08 19:00:30,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb0-1678302020890-1678302030950 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:30,965 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb0-1678302020890-1678302030950 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1678302020890 2023-03-08 19:00:30,965 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb0-1678302020890-1678302030950 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-03-08 19:00:30,965 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-03-08 19:00:30,978 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1678302020890-1678302030950 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } 2023-03-08 19:00:30,980 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:30,980 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:30,980 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:30,980 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:30,980 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:30,980 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:30,980 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:30,980 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:30,981 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:30,981 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:30,981 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:30,981 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:30,981 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:30,981 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:30,984 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/172aadf8677145f0b28293dbaeda5b9f, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/9549786755914507847487aebb599d51] hfiles 2023-03-08 19:00:30,984 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/172aadf8677145f0b28293dbaeda5b9f 2023-03-08 19:00:30,984 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/9549786755914507847487aebb599d51 2023-03-08 19:00:30,986 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/7e72d8ad17d24d9f8bfdc8e8a32a99ed, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/bd07ca0c4ad54ba0bb89f6a8507d4e9b] hfiles 2023-03-08 19:00:30,986 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/7e72d8ad17d24d9f8bfdc8e8a32a99ed 2023-03-08 19:00:30,986 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/bd07ca0c4ad54ba0bb89f6a8507d4e9b 2023-03-08 19:00:30,987 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/657092e48c0b4ad0ab0fa0681948dafb, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/f24cdc04e3d949cf8fcfbd4389e9cdfb] hfiles 2023-03-08 19:00:30,987 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/657092e48c0b4ad0ab0fa0681948dafb 2023-03-08 19:00:30,987 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/f24cdc04e3d949cf8fcfbd4389e9cdfb 2023-03-08 19:00:30,987 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/2a0e9bd776cc4f468bd20a74e4f61006, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/5b72a399cd494fa4836e4a3e20c6b1ba] hfiles 2023-03-08 19:00:30,988 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/2a0e9bd776cc4f468bd20a74e4f61006 2023-03-08 19:00:30,988 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/5b72a399cd494fa4836e4a3e20c6b1ba 2023-03-08 19:00:30,989 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/4943e5745d97426d8d62901400379e84, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/57dba886df0248f48f93263d56e12c12] hfiles 2023-03-08 19:00:30,989 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/4943e5745d97426d8d62901400379e84 2023-03-08 19:00:30,989 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/57dba886df0248f48f93263d56e12c12 2023-03-08 19:00:30,990 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/64a5b55bb32e4bff96aba52c33665694, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/bc3874f6411e445bb582265e908e860e] hfiles 2023-03-08 19:00:30,990 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/64a5b55bb32e4bff96aba52c33665694 2023-03-08 19:00:30,990 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/bc3874f6411e445bb582265e908e860e 2023-03-08 19:00:30,993 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf] hfiles 2023-03-08 19:00:30,993 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 2023-03-08 19:00:30,993 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 2023-03-08 19:00:30,993 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 2023-03-08 19:00:30,993 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 2023-03-08 19:00:30,993 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 2023-03-08 19:00:30,994 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc 2023-03-08 19:00:30,994 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d 2023-03-08 19:00:30,994 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 2023-03-08 19:00:30,994 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 2023-03-08 19:00:30,994 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 2023-03-08 19:00:30,994 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 2023-03-08 19:00:30,994 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf 2023-03-08 19:00:31,215 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-03-08 19:00:31,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1678302020890-1678302030950 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } is done 2023-03-08 19:00:31,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snaptb0-1678302020890-1678302030950 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 }' is still in progress! 2023-03-08 19:00:31,217 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-03-08 19:00:31,431 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb0-1678302020890-1678302030950 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } as finished. 2023-03-08 19:00:31,432 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb0-1678302020890-1678302030950 2023-03-08 19:00:31,432 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1678302020890-1678302030950 2023-03-08 19:00:31,457 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1678302020890-1678302030950 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/hbase-failsafe-snaptb0-1678302020890-1678302030950 2023-03-08 19:00:31,477 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb0-1678302020890-1678302030950 of table testRestoreSnapshot-1678302020890 completed 2023-03-08 19:00:31,478 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1678302020890-1678302030950 2023-03-08 19:00:31,478 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1678302020890-1678302030950 2023-03-08 19:00:31,478 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb0-1678302020890-1678302030950 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1678302020890 at 1678302030965Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1678302020890-1678302030950 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } at 1678302030978 (+13 ms)Consolidate snapshot: hbase-failsafe-snaptb0-1678302020890-1678302030950 at 1678302031432 (+454 ms)Loading Region manifests for hbase-failsafe-snaptb0-1678302020890-1678302030950 at 1678302031432Writing data manifest for hbase-failsafe-snaptb0-1678302020890-1678302030950 at 1678302031436 (+4 ms)Verifying snapshot: hbase-failsafe-snaptb0-1678302020890-1678302030950 at 1678302031445 (+9 ms)Snapshot hbase-failsafe-snaptb0-1678302020890-1678302030950 of table testRestoreSnapshot-1678302020890 completed at 1678302031477 (+32 ms) 2023-03-08 19:00:31,479 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=90, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:31,481 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=90, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE in 516 msec 2023-03-08 19:00:31,717 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-03-08 19:00:31,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1678302020890-1678302030950 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } is done 2023-03-08 19:00:31,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb0-1678302020890-1678302030950 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 }' has completed, notifying client. 2023-03-08 19:00:31,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1678302020890 snapshot=name: "snaptb0-1678302020890" table: "testRestoreSnapshot-1678302020890" creation_time: 1678302024511 type: DISABLED version: 2 ttl: 0 ) 2023-03-08 19:00:31,743 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(880): Restore snapshot=snaptb0-1678302020890 as table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:31,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-03-08 19:00:31,761 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/.tabledesc/.tableinfo.0000000002 2023-03-08 19:00:31,762 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/.tabledesc/.tableinfo.0000000001 2023-03-08 19:00:31,763 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/.tabledesc/.tableinfo.0000000002 2023-03-08 19:00:31,764 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb0-1678302020890 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } 2023-03-08 19:00:31,767 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb0-1678302020890" table: "testRestoreSnapshot-1678302020890" creation_time: 1678302024511 type: DISABLED version: 2 ttl: 0 2023-03-08 19:00:31,768 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890 2023-03-08 19:00:31,777 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:31,785 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 2023-03-08 19:00:31,786 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d 2023-03-08 19:00:31,787 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 2023-03-08 19:00:31,789 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 2023-03-08 19:00:31,790 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 2023-03-08 19:00:31,791 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 2023-03-08 19:00:31,791 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:31,791 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:31,792 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:31,792 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:31,792 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:31,792 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:31,797 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/5b72a399cd494fa4836e4a3e20c6b1ba to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/5b72a399cd494fa4836e4a3e20c6b1ba 2023-03-08 19:00:31,798 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/64a5b55bb32e4bff96aba52c33665694 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/64a5b55bb32e4bff96aba52c33665694 2023-03-08 19:00:31,798 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/bd07ca0c4ad54ba0bb89f6a8507d4e9b to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/bd07ca0c4ad54ba0bb89f6a8507d4e9b 2023-03-08 19:00:31,798 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/4943e5745d97426d8d62901400379e84 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/4943e5745d97426d8d62901400379e84 2023-03-08 19:00:31,798 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/657092e48c0b4ad0ab0fa0681948dafb to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/657092e48c0b4ad0ab0fa0681948dafb 2023-03-08 19:00:31,799 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/172aadf8677145f0b28293dbaeda5b9f to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/172aadf8677145f0b28293dbaeda5b9f 2023-03-08 19:00:31,799 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb0-1678302020890" table: "testRestoreSnapshot-1678302020890" creation_time: 1678302024511 type: DISABLED version: 2 ttl: 0 2023-03-08 19:00:31,801 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302031800"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:31,801 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302031800"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:31,801 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302031800"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:31,801 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302031800"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:31,801 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302031800"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:31,801 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302031800"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:31,805 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-03-08 19:00:31,805 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'}] 2023-03-08 19:00:31,806 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302031801"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302031801"}]},"ts":"1678302031801"} 2023-03-08 19:00:31,806 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302031801"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302031801"}]},"ts":"1678302031801"} 2023-03-08 19:00:31,806 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302031801"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302031801"}]},"ts":"1678302031801"} 2023-03-08 19:00:31,806 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302031801"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302031801"}]},"ts":"1678302031801"} 2023-03-08 19:00:31,806 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302031801"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302031801"}]},"ts":"1678302031801"} 2023-03-08 19:00:31,806 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302031801"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302031801"}]},"ts":"1678302031801"} 2023-03-08 19:00:31,808 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-03-08 19:00:31,808 INFO [PEWorker-1] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-03-08 19:00:31,808 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'}] 2023-03-08 19:00:31,809 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb0-1678302020890 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1678302020890 completed! 2023-03-08 19:00:31,811 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1678302020890 snapshot=name: "snaptb0-1678302020890" table: "testRestoreSnapshot-1678302020890" creation_time: 1678302024511 type: DISABLED version: 2 ttl: 0 ) in 76 msec 2023-03-08 19:00:31,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-03-08 19:00:31,999 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1678302020890, procId: 91 completed 2023-03-08 19:00:31,999 INFO [Listener at localhost/45021] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb0-1678302020890-1678302030950 2023-03-08 19:00:32,003 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(748): Client=jenkins//172.31.2.20 delete name: "hbase-failsafe-snaptb0-1678302020890-1678302030950" 2023-03-08 19:00:32,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb0-1678302020890-1678302030950 2023-03-08 19:00:32,007 INFO [Listener at localhost/45021] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1678302020890 2023-03-08 19:00:32,007 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$10(2385): Client=jenkins//172.31.2.20 enable testRestoreSnapshot-1678302020890 2023-03-08 19:00:32,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:32,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-03-08 19:00:32,011 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1678302020890 2023-03-08 19:00:32,011 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302032011"}]},"ts":"1678302032011"} 2023-03-08 19:00:32,013 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=ENABLING in hbase:meta 2023-03-08 19:00:32,015 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-03-08 19:00:32,018 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN}, {pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN}, {pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN}, {pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN}, {pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN}, {pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN}] 2023-03-08 19:00:32,019 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN 2023-03-08 19:00:32,019 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN 2023-03-08 19:00:32,020 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN 2023-03-08 19:00:32,020 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN 2023-03-08 19:00:32,020 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN 2023-03-08 19:00:32,020 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:32,020 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:32,020 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:32,021 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:32,021 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:32,021 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN 2023-03-08 19:00:32,022 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:32,171 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase6.apache.org=0} racks are {/default-rack=0} 2023-03-08 19:00:32,171 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-08 19:00:32,171 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-03-08 19:00:32,171 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-03-08 19:00:32,171 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-03-08 19:00:32,174 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:32,174 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:32,174 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302032174"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302032174"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302032174"}]},"ts":"1678302032174"} 2023-03-08 19:00:32,174 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:32,174 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302032174"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302032174"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302032174"}]},"ts":"1678302032174"} 2023-03-08 19:00:32,175 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302032174"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302032174"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302032174"}]},"ts":"1678302032174"} 2023-03-08 19:00:32,174 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:32,174 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:32,175 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302032174"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302032174"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302032174"}]},"ts":"1678302032174"} 2023-03-08 19:00:32,175 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302032174"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302032174"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302032174"}]},"ts":"1678302032174"} 2023-03-08 19:00:32,176 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=98, state=RUNNABLE; OpenRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:32,177 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=97, state=RUNNABLE; OpenRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:32,177 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=94, state=RUNNABLE; OpenRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:32,178 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=96, state=RUNNABLE; OpenRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:32,179 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=95, state=RUNNABLE; OpenRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:32,180 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:32,180 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302032180"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302032180"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302032180"}]},"ts":"1678302032180"} 2023-03-08 19:00:32,182 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=93, state=RUNNABLE; OpenRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:32,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-03-08 19:00:32,331 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:32,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''} 2023-03-08 19:00:32,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:32,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:32,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:32,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:32,332 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:32,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'} 2023-03-08 19:00:32,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:32,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:32,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:32,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:32,334 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:32,334 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:32,334 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:32,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'} 2023-03-08 19:00:32,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:32,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:32,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:32,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:32,335 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf 2023-03-08 19:00:32,335 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf 2023-03-08 19:00:32,335 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf 2023-03-08 19:00:32,335 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf 2023-03-08 19:00:32,335 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ea43d1afb394fd7a05814d2275bdeffd columnFamilyName cf 2023-03-08 19:00:32,335 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e3eaed3ac05b4e65948c1d4b5c75462d columnFamilyName cf 2023-03-08 19:00:32,336 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:32,337 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf 2023-03-08 19:00:32,337 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf 2023-03-08 19:00:32,337 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 668bd868cc8589cf26791b7dda1212d8 columnFamilyName cf 2023-03-08 19:00:32,341 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/7e72d8ad17d24d9f8bfdc8e8a32a99ed 2023-03-08 19:00:32,341 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(310): Store=ea43d1afb394fd7a05814d2275bdeffd/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:32,341 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/bc3874f6411e445bb582265e908e860e 2023-03-08 19:00:32,342 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(310): Store=e3eaed3ac05b4e65948c1d4b5c75462d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:32,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:32,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:32,343 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/2a0e9bd776cc4f468bd20a74e4f61006 2023-03-08 19:00:32,343 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(310): Store=668bd868cc8589cf26791b7dda1212d8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:32,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:32,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:32,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:32,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:32,347 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:32,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:32,349 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened ea43d1afb394fd7a05814d2275bdeffd; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72053227, jitterRate=0.07367675006389618}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:32,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for ea43d1afb394fd7a05814d2275bdeffd: 2023-03-08 19:00:32,349 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened e3eaed3ac05b4e65948c1d4b5c75462d; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63159451, jitterRate=-0.058850839734077454}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:32,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:32,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for e3eaed3ac05b4e65948c1d4b5c75462d: 2023-03-08 19:00:32,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd., pid=100, masterSystemTime=1678302032329 2023-03-08 19:00:32,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d., pid=99, masterSystemTime=1678302032328 2023-03-08 19:00:32,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 668bd868cc8589cf26791b7dda1212d8; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72868186, jitterRate=0.08582058548927307}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:32,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 668bd868cc8589cf26791b7dda1212d8: 2023-03-08 19:00:32,351 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8., pid=104, masterSystemTime=1678302032331 2023-03-08 19:00:32,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:32,352 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:32,352 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:32,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'} 2023-03-08 19:00:32,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:32,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:32,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:32,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:32,353 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:32,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:32,353 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:32,353 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:32,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'} 2023-03-08 19:00:32,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:32,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:32,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:32,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:32,354 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:32,353 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302032353"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302032353"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302032353"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302032353"}]},"ts":"1678302032353"} 2023-03-08 19:00:32,355 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:32,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:32,355 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:32,355 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:32,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'} 2023-03-08 19:00:32,355 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302032355"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302032355"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302032355"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302032355"}]},"ts":"1678302032355"} 2023-03-08 19:00:32,356 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf 2023-03-08 19:00:32,356 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf 2023-03-08 19:00:32,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:32,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:32,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:32,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:32,356 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7cdf03195577655a4e5d1e7f4ff92392 columnFamilyName cf 2023-03-08 19:00:32,356 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:32,356 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302032356"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302032356"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302032356"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302032356"}]},"ts":"1678302032356"} 2023-03-08 19:00:32,358 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:32,360 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf 2023-03-08 19:00:32,360 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=98 2023-03-08 19:00:32,361 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=98, state=SUCCESS; OpenRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,46129,1678302018069 in 181 msec 2023-03-08 19:00:32,360 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:32,361 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf 2023-03-08 19:00:32,361 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region be16768f5782c50d4c94d5fdc74d3f28 columnFamilyName cf 2023-03-08 19:00:32,362 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf 2023-03-08 19:00:32,362 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf 2023-03-08 19:00:32,363 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=97 2023-03-08 19:00:32,363 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=97, state=SUCCESS; OpenRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,41161,1678302018011 in 182 msec 2023-03-08 19:00:32,363 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN in 342 msec 2023-03-08 19:00:32,363 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b8add49535b4b0366d051f3a48b2a333 columnFamilyName cf 2023-03-08 19:00:32,364 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=93 2023-03-08 19:00:32,364 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=93, state=SUCCESS; OpenRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939 in 178 msec 2023-03-08 19:00:32,365 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN in 345 msec 2023-03-08 19:00:32,365 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN in 348 msec 2023-03-08 19:00:32,366 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/f24cdc04e3d949cf8fcfbd4389e9cdfb 2023-03-08 19:00:32,366 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(310): Store=7cdf03195577655a4e5d1e7f4ff92392/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:32,367 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:32,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:32,369 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/57dba886df0248f48f93263d56e12c12 2023-03-08 19:00:32,369 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(310): Store=be16768f5782c50d4c94d5fdc74d3f28/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:32,370 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/9549786755914507847487aebb599d51 2023-03-08 19:00:32,370 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(310): Store=b8add49535b4b0366d051f3a48b2a333/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:32,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:32,371 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:32,371 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:32,371 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:32,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:32,372 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 7cdf03195577655a4e5d1e7f4ff92392; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62389678, jitterRate=-0.07032135128974915}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:32,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 7cdf03195577655a4e5d1e7f4ff92392: 2023-03-08 19:00:32,373 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392., pid=103, masterSystemTime=1678302032328 2023-03-08 19:00:32,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:32,375 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:32,375 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:32,375 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302032375"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302032375"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302032375"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302032375"}]},"ts":"1678302032375"} 2023-03-08 19:00:32,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:32,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:32,377 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened b8add49535b4b0366d051f3a48b2a333; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75308523, jitterRate=0.12218444049358368}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:32,377 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened be16768f5782c50d4c94d5fdc74d3f28; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73297722, jitterRate=0.09222117066383362}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:32,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for be16768f5782c50d4c94d5fdc74d3f28: 2023-03-08 19:00:32,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for b8add49535b4b0366d051f3a48b2a333: 2023-03-08 19:00:32,378 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28., pid=101, masterSystemTime=1678302032329 2023-03-08 19:00:32,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333., pid=102, masterSystemTime=1678302032331 2023-03-08 19:00:32,380 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=95 2023-03-08 19:00:32,380 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=95, state=SUCCESS; OpenRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,46129,1678302018069 in 198 msec 2023-03-08 19:00:32,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:32,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:32,381 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:32,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:32,381 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302032381"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302032381"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302032381"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302032381"}]},"ts":"1678302032381"} 2023-03-08 19:00:32,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:32,382 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=95, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN in 362 msec 2023-03-08 19:00:32,382 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:32,382 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302032382"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302032382"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302032382"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302032382"}]},"ts":"1678302032382"} 2023-03-08 19:00:32,385 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=94 2023-03-08 19:00:32,385 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=94, state=SUCCESS; OpenRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,41161,1678302018011 in 206 msec 2023-03-08 19:00:32,386 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=96 2023-03-08 19:00:32,386 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN in 369 msec 2023-03-08 19:00:32,386 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=96, state=SUCCESS; OpenRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,33875,1678302017939 in 206 msec 2023-03-08 19:00:32,388 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=96, resume processing ppid=92 2023-03-08 19:00:32,388 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN in 368 msec 2023-03-08 19:00:32,389 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302032389"}]},"ts":"1678302032389"} 2023-03-08 19:00:32,390 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=ENABLED in hbase:meta 2023-03-08 19:00:32,392 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1678302020890' was successfully enabled. 2023-03-08 19:00:32,394 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1678302020890 in 386 msec 2023-03-08 19:00:32,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-03-08 19:00:32,763 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1678302020890, procId: 92 completed 2023-03-08 19:00:32,763 DEBUG [Listener at localhost/45021] zookeeper.ReadOnlyZKClient(139): Connect 0x29e7c4cf to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:32,769 DEBUG [Listener at localhost/45021] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e907963, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:32,770 DEBUG [hconnection-0x116b3a18-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:32,772 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:57954, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:32,840 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-03-08 19:00:32,844 DEBUG [hconnection-0x116b3a18-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:32,847 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:55080, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:32,899 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7b77b3f7 to 127.0.0.1:59169 2023-03-08 19:00:32,899 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:32,990 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-03-08 19:00:32,993 DEBUG [hconnection-0x116b3a18-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:32,996 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47276, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:33,085 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-03-08 19:00:33,166 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-03-08 19:00:33,279 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-03-08 19:00:34,163 INFO [Listener at localhost/45021] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1678302020890 2023-03-08 19:00:34,164 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$11(2418): Client=jenkins//172.31.2.20 disable testRestoreSnapshot-1678302020890 2023-03-08 19:00:34,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:34,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-03-08 19:00:34,168 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302034168"}]},"ts":"1678302034168"} 2023-03-08 19:00:34,169 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=DISABLING in hbase:meta 2023-03-08 19:00:34,171 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1678302020890 to state=DISABLING 2023-03-08 19:00:34,172 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN}, {pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN}, {pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN}, {pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN}, {pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN}, {pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN}] 2023-03-08 19:00:34,175 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN 2023-03-08 19:00:34,175 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN 2023-03-08 19:00:34,175 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN 2023-03-08 19:00:34,176 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN 2023-03-08 19:00:34,176 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN 2023-03-08 19:00:34,176 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:34,176 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:34,176 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302034176"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302034176"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302034176"}]},"ts":"1678302034176"} 2023-03-08 19:00:34,176 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302034176"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302034176"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302034176"}]},"ts":"1678302034176"} 2023-03-08 19:00:34,176 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:34,176 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:34,176 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:34,177 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302034176"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302034176"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302034176"}]},"ts":"1678302034176"} 2023-03-08 19:00:34,177 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302034176"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302034176"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302034176"}]},"ts":"1678302034176"} 2023-03-08 19:00:34,177 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302034176"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302034176"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302034176"}]},"ts":"1678302034176"} 2023-03-08 19:00:34,178 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=106, state=RUNNABLE; CloseRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:34,180 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=109, state=RUNNABLE; CloseRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:34,181 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=111, state=RUNNABLE; CloseRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:34,183 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=108, state=RUNNABLE; CloseRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:34,183 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN 2023-03-08 19:00:34,184 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=107, state=RUNNABLE; CloseRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:34,184 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:34,184 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302034184"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302034184"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302034184"}]},"ts":"1678302034184"} 2023-03-08 19:00:34,186 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=110, state=RUNNABLE; CloseRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:34,332 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:34,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing b8add49535b4b0366d051f3a48b2a333, disabling compactions & flushes 2023-03-08 19:00:34,333 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:34,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:34,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. after waiting 0 ms 2023-03-08 19:00:34,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:34,337 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:34,339 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 7cdf03195577655a4e5d1e7f4ff92392, disabling compactions & flushes 2023-03-08 19:00:34,339 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:34,339 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:34,339 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. after waiting 0 ms 2023-03-08 19:00:34,339 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:34,339 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:34,339 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing be16768f5782c50d4c94d5fdc74d3f28, disabling compactions & flushes 2023-03-08 19:00:34,339 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:34,339 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:34,339 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. after waiting 0 ms 2023-03-08 19:00:34,339 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:34,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-03-08 19:00:34,346 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:34,346 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-03-08 19:00:34,346 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for b8add49535b4b0366d051f3a48b2a333: 2023-03-08 19:00:34,346 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:34,346 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 7cdf03195577655a4e5d1e7f4ff92392: 2023-03-08 19:00:34,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-03-08 19:00:34,347 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:34,347 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:34,347 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:34,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for be16768f5782c50d4c94d5fdc74d3f28: 2023-03-08 19:00:34,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 668bd868cc8589cf26791b7dda1212d8, disabling compactions & flushes 2023-03-08 19:00:34,348 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:34,348 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:34,348 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. after waiting 0 ms 2023-03-08 19:00:34,348 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:34,348 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=CLOSED 2023-03-08 19:00:34,348 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302034348"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302034348"}]},"ts":"1678302034348"} 2023-03-08 19:00:34,348 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:34,348 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:34,348 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing e3eaed3ac05b4e65948c1d4b5c75462d, disabling compactions & flushes 2023-03-08 19:00:34,348 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:34,348 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:34,348 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. after waiting 0 ms 2023-03-08 19:00:34,348 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:34,349 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=CLOSED 2023-03-08 19:00:34,349 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302034349"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302034349"}]},"ts":"1678302034349"} 2023-03-08 19:00:34,349 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:34,350 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:34,350 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing ea43d1afb394fd7a05814d2275bdeffd, disabling compactions & flushes 2023-03-08 19:00:34,350 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:34,350 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:34,350 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. after waiting 0 ms 2023-03-08 19:00:34,350 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:34,351 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=CLOSED 2023-03-08 19:00:34,351 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302034351"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302034351"}]},"ts":"1678302034351"} 2023-03-08 19:00:34,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-03-08 19:00:34,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-03-08 19:00:34,353 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:34,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 668bd868cc8589cf26791b7dda1212d8: 2023-03-08 19:00:34,353 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:34,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for e3eaed3ac05b4e65948c1d4b5c75462d: 2023-03-08 19:00:34,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-03-08 19:00:34,354 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:34,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for ea43d1afb394fd7a05814d2275bdeffd: 2023-03-08 19:00:34,355 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:34,356 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=109 2023-03-08 19:00:34,356 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=109, state=SUCCESS; CloseRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,33875,1678302017939 in 172 msec 2023-03-08 19:00:34,356 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=CLOSED 2023-03-08 19:00:34,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:34,357 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302034356"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302034356"}]},"ts":"1678302034356"} 2023-03-08 19:00:34,357 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=CLOSED 2023-03-08 19:00:34,357 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=108 2023-03-08 19:00:34,357 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302034357"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302034357"}]},"ts":"1678302034357"} 2023-03-08 19:00:34,357 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=108, state=SUCCESS; CloseRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,46129,1678302018069 in 168 msec 2023-03-08 19:00:34,357 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN in 184 msec 2023-03-08 19:00:34,357 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=107 2023-03-08 19:00:34,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:34,357 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=107, state=SUCCESS; CloseRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,41161,1678302018011 in 171 msec 2023-03-08 19:00:34,358 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=CLOSED 2023-03-08 19:00:34,358 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302034358"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302034358"}]},"ts":"1678302034358"} 2023-03-08 19:00:34,359 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN in 185 msec 2023-03-08 19:00:34,359 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN in 186 msec 2023-03-08 19:00:34,363 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=106 2023-03-08 19:00:34,364 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=106, state=SUCCESS; CloseRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939 in 180 msec 2023-03-08 19:00:34,364 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=111 2023-03-08 19:00:34,364 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=111, state=SUCCESS; CloseRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,46129,1678302018069 in 178 msec 2023-03-08 19:00:34,365 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN in 191 msec 2023-03-08 19:00:34,365 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=110 2023-03-08 19:00:34,365 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=110, state=SUCCESS; CloseRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,41161,1678302018011 in 177 msec 2023-03-08 19:00:34,365 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN in 192 msec 2023-03-08 19:00:34,366 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=110, resume processing ppid=105 2023-03-08 19:00:34,366 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN in 193 msec 2023-03-08 19:00:34,367 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302034367"}]},"ts":"1678302034367"} 2023-03-08 19:00:34,368 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=DISABLED in hbase:meta 2023-03-08 19:00:34,370 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1678302020890 to state=DISABLED 2023-03-08 19:00:34,372 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1678302020890 in 207 msec 2023-03-08 19:00:34,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-03-08 19:00:34,419 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1678302020890, procId: 105 completed 2023-03-08 19:00:34,427 INFO [Listener at localhost/45021] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1678302020890-1678302034427 2023-03-08 19:00:34,428 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1580): Client=jenkins//172.31.2.20 snapshot request for:{ ss=hbase-failsafe-emptySnaptb-1678302020890-1678302034427 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:34,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1678302034428 (current time:1678302034428). 2023-03-08 19:00:34,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-03-08 19:00:34,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] zookeeper.ReadOnlyZKClient(139): Connect 0x73b31a4a to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:34,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5258589c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:34,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:34,437 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:57970, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:34,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73b31a4a to 127.0.0.1:59169 2023-03-08 19:00:34,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:34,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-03-08 19:00:34,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-emptySnaptb-1678302020890-1678302034427 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:34,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:34,441 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:34,442 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE; timeout=600000, timestamp=1678302634442 2023-03-08 19:00:34,442 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1678302020890-1678302034427 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1678302020890 2023-03-08 19:00:34,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-emptySnaptb-1678302020890-1678302034427 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:34,442 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-emptySnaptb-1678302020890-1678302034427 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-03-08 19:00:34,442 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-03-08 19:00:34,457 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1678302020890-1678302034427 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } 2023-03-08 19:00:34,459 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:34,459 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:34,459 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:34,459 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:34,459 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:34,459 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:34,459 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:34,459 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:34,460 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:34,460 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:34,460 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:34,460 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:34,460 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:34,460 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:34,463 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/57dba886df0248f48f93263d56e12c12] hfiles 2023-03-08 19:00:34,464 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/57dba886df0248f48f93263d56e12c12 2023-03-08 19:00:34,464 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/7e72d8ad17d24d9f8bfdc8e8a32a99ed] hfiles 2023-03-08 19:00:34,464 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/7e72d8ad17d24d9f8bfdc8e8a32a99ed 2023-03-08 19:00:34,464 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/2a0e9bd776cc4f468bd20a74e4f61006] hfiles 2023-03-08 19:00:34,464 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/2a0e9bd776cc4f468bd20a74e4f61006 2023-03-08 19:00:34,464 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/f24cdc04e3d949cf8fcfbd4389e9cdfb] hfiles 2023-03-08 19:00:34,465 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/f24cdc04e3d949cf8fcfbd4389e9cdfb 2023-03-08 19:00:34,467 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/9549786755914507847487aebb599d51] hfiles 2023-03-08 19:00:34,467 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/9549786755914507847487aebb599d51 2023-03-08 19:00:34,469 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf] hfiles 2023-03-08 19:00:34,469 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 2023-03-08 19:00:34,469 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 2023-03-08 19:00:34,469 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc 2023-03-08 19:00:34,469 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 2023-03-08 19:00:34,469 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 2023-03-08 19:00:34,469 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf 2023-03-08 19:00:34,470 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/bc3874f6411e445bb582265e908e860e] hfiles 2023-03-08 19:00:34,470 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/bc3874f6411e445bb582265e908e860e 2023-03-08 19:00:34,507 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-emptySnaptb-1678302020890-1678302034427 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } as finished. 2023-03-08 19:00:34,507 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-emptySnaptb-1678302020890-1678302034427 2023-03-08 19:00:34,507 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1678302020890-1678302034427 2023-03-08 19:00:34,526 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1678302020890-1678302034427 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/hbase-failsafe-emptySnaptb-1678302020890-1678302034427 2023-03-08 19:00:34,693 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-03-08 19:00:34,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1678302020890-1678302034427 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } is done 2023-03-08 19:00:34,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-emptySnaptb-1678302020890-1678302034427 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 }' is still in progress! 2023-03-08 19:00:34,694 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-03-08 19:00:34,945 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-emptySnaptb-1678302020890-1678302034427 of table testRestoreSnapshot-1678302020890 completed 2023-03-08 19:00:34,945 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1678302020890-1678302034427 2023-03-08 19:00:34,945 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1678302020890-1678302034427 2023-03-08 19:00:34,945 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1678302020890-1678302034427 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1678302020890 at 1678302034442Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1678302020890-1678302034427 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } at 1678302034457 (+15 ms)Consolidate snapshot: hbase-failsafe-emptySnaptb-1678302020890-1678302034427 at 1678302034507 (+50 ms)Loading Region manifests for hbase-failsafe-emptySnaptb-1678302020890-1678302034427 at 1678302034507Writing data manifest for hbase-failsafe-emptySnaptb-1678302020890-1678302034427 at 1678302034510 (+3 ms)Verifying snapshot: hbase-failsafe-emptySnaptb-1678302020890-1678302034427 at 1678302034519 (+9 ms)Snapshot hbase-failsafe-emptySnaptb-1678302020890-1678302034427 of table testRestoreSnapshot-1678302020890 completed at 1678302034945 (+426 ms) 2023-03-08 19:00:34,947 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:34,948 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE in 507 msec 2023-03-08 19:00:35,194 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-03-08 19:00:35,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1678302020890-1678302034427 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } is done 2023-03-08 19:00:35,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-emptySnaptb-1678302020890-1678302034427 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 }' has completed, notifying client. 2023-03-08 19:00:35,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1678302020890 snapshot=name: "emptySnaptb-1678302020890" table: "testRestoreSnapshot-1678302020890" creation_time: 1678302022548 type: DISABLED version: 2 ttl: 0 ) 2023-03-08 19:00:35,200 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(880): Restore snapshot=emptySnaptb-1678302020890 as table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:35,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-03-08 19:00:35,213 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/.tabledesc/.tableinfo.0000000003 2023-03-08 19:00:35,214 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/.tabledesc/.tableinfo.0000000002 2023-03-08 19:00:35,214 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/.tabledesc/.tableinfo.0000000003 2023-03-08 19:00:35,215 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=emptySnaptb-1678302020890 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } 2023-03-08 19:00:35,216 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "emptySnaptb-1678302020890" table: "testRestoreSnapshot-1678302020890" creation_time: 1678302022548 type: DISABLED version: 2 ttl: 0 2023-03-08 19:00:35,216 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890 2023-03-08 19:00:35,223 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:35,226 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 2023-03-08 19:00:35,227 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 2023-03-08 19:00:35,228 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc 2023-03-08 19:00:35,230 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 2023-03-08 19:00:35,231 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 2023-03-08 19:00:35,232 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf 2023-03-08 19:00:35,232 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:35,232 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:35,232 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:35,232 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:35,233 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:35,233 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:35,238 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/f24cdc04e3d949cf8fcfbd4389e9cdfb to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/f24cdc04e3d949cf8fcfbd4389e9cdfb 2023-03-08 19:00:35,238 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/2a0e9bd776cc4f468bd20a74e4f61006 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/2a0e9bd776cc4f468bd20a74e4f61006 2023-03-08 19:00:35,238 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/9549786755914507847487aebb599d51 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/9549786755914507847487aebb599d51 2023-03-08 19:00:35,238 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/57dba886df0248f48f93263d56e12c12 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/57dba886df0248f48f93263d56e12c12 2023-03-08 19:00:35,239 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/7e72d8ad17d24d9f8bfdc8e8a32a99ed to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/7e72d8ad17d24d9f8bfdc8e8a32a99ed 2023-03-08 19:00:35,239 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/bc3874f6411e445bb582265e908e860e to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/bc3874f6411e445bb582265e908e860e 2023-03-08 19:00:35,239 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "emptySnaptb-1678302020890" table: "testRestoreSnapshot-1678302020890" creation_time: 1678302022548 type: DISABLED version: 2 ttl: 0 2023-03-08 19:00:35,241 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302035240"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:35,241 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302035240"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:35,241 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302035240"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:35,241 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302035240"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:35,241 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302035240"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:35,241 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302035240"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:35,243 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-03-08 19:00:35,243 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'}] 2023-03-08 19:00:35,243 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302035241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302035241"}]},"ts":"1678302035241"} 2023-03-08 19:00:35,244 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302035241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302035241"}]},"ts":"1678302035241"} 2023-03-08 19:00:35,244 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302035241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302035241"}]},"ts":"1678302035241"} 2023-03-08 19:00:35,244 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302035241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302035241"}]},"ts":"1678302035241"} 2023-03-08 19:00:35,244 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302035241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302035241"}]},"ts":"1678302035241"} 2023-03-08 19:00:35,244 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302035241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302035241"}]},"ts":"1678302035241"} 2023-03-08 19:00:35,246 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-03-08 19:00:35,246 INFO [PEWorker-2] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-03-08 19:00:35,246 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'}] 2023-03-08 19:00:35,246 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=emptySnaptb-1678302020890 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1678302020890 completed! 2023-03-08 19:00:35,248 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1678302020890 snapshot=name: "emptySnaptb-1678302020890" table: "testRestoreSnapshot-1678302020890" creation_time: 1678302022548 type: DISABLED version: 2 ttl: 0 ) in 48 msec 2023-03-08 19:00:35,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-03-08 19:00:35,454 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1678302020890, procId: 119 completed 2023-03-08 19:00:35,454 INFO [Listener at localhost/45021] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1678302020890-1678302034427 2023-03-08 19:00:35,454 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(748): Client=jenkins//172.31.2.20 delete name: "hbase-failsafe-emptySnaptb-1678302020890-1678302034427" 2023-03-08 19:00:35,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-emptySnaptb-1678302020890-1678302034427 2023-03-08 19:00:35,457 INFO [Listener at localhost/45021] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1678302020890 2023-03-08 19:00:35,458 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$10(2385): Client=jenkins//172.31.2.20 enable testRestoreSnapshot-1678302020890 2023-03-08 19:00:35,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:35,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-03-08 19:00:35,461 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1678302020890 2023-03-08 19:00:35,461 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302035461"}]},"ts":"1678302035461"} 2023-03-08 19:00:35,462 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=ENABLING in hbase:meta 2023-03-08 19:00:35,465 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-03-08 19:00:35,465 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN}, {pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN}, {pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN}, {pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN}, {pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN}, {pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN}] 2023-03-08 19:00:35,466 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN 2023-03-08 19:00:35,466 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN 2023-03-08 19:00:35,466 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN 2023-03-08 19:00:35,467 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN 2023-03-08 19:00:35,467 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN 2023-03-08 19:00:35,467 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:35,467 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:35,467 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:35,468 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:35,468 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN 2023-03-08 19:00:35,468 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:35,468 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:35,618 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase6.apache.org=0} racks are {/default-rack=0} 2023-03-08 19:00:35,618 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-08 19:00:35,618 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-03-08 19:00:35,618 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-03-08 19:00:35,618 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-03-08 19:00:35,622 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:35,622 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:35,622 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302035622"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302035622"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302035622"}]},"ts":"1678302035622"} 2023-03-08 19:00:35,622 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:35,622 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:35,623 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302035622"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302035622"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302035622"}]},"ts":"1678302035622"} 2023-03-08 19:00:35,622 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:35,623 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302035622"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302035622"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302035622"}]},"ts":"1678302035622"} 2023-03-08 19:00:35,623 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302035622"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302035622"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302035622"}]},"ts":"1678302035622"} 2023-03-08 19:00:35,623 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302035622"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302035622"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302035622"}]},"ts":"1678302035622"} 2023-03-08 19:00:35,624 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=123, state=RUNNABLE; OpenRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:35,624 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=128, ppid=125, state=RUNNABLE; OpenRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:35,625 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=124, state=RUNNABLE; OpenRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:35,626 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=126, state=RUNNABLE; OpenRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:35,627 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:35,627 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302035627"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302035627"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302035627"}]},"ts":"1678302035627"} 2023-03-08 19:00:35,627 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=122, state=RUNNABLE; OpenRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:35,629 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=121, state=RUNNABLE; OpenRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:35,636 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-08 19:00:35,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-03-08 19:00:35,778 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:35,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'} 2023-03-08 19:00:35,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:35,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:35,778 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:35,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:35,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:35,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'} 2023-03-08 19:00:35,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:35,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:35,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:35,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:35,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:35,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'} 2023-03-08 19:00:35,780 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:35,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:35,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:35,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:35,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:35,780 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:35,781 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf 2023-03-08 19:00:35,781 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf 2023-03-08 19:00:35,781 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:35,781 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7cdf03195577655a4e5d1e7f4ff92392 columnFamilyName cf 2023-03-08 19:00:35,781 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf 2023-03-08 19:00:35,782 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf 2023-03-08 19:00:35,782 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region be16768f5782c50d4c94d5fdc74d3f28 columnFamilyName cf 2023-03-08 19:00:35,782 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf 2023-03-08 19:00:35,782 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf 2023-03-08 19:00:35,782 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 668bd868cc8589cf26791b7dda1212d8 columnFamilyName cf 2023-03-08 19:00:35,783 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(310): Store=7cdf03195577655a4e5d1e7f4ff92392/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:35,783 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(310): Store=be16768f5782c50d4c94d5fdc74d3f28/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:35,783 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(310): Store=668bd868cc8589cf26791b7dda1212d8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:35,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:35,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:35,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:35,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:35,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:35,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:35,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:35,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:35,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:35,787 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened be16768f5782c50d4c94d5fdc74d3f28; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64053681, jitterRate=-0.045525774359703064}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:35,787 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 7cdf03195577655a4e5d1e7f4ff92392; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73255742, jitterRate=0.0915956199169159}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:35,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for be16768f5782c50d4c94d5fdc74d3f28: 2023-03-08 19:00:35,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 7cdf03195577655a4e5d1e7f4ff92392: 2023-03-08 19:00:35,788 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 668bd868cc8589cf26791b7dda1212d8; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67829706, jitterRate=0.010741382837295532}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:35,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 668bd868cc8589cf26791b7dda1212d8: 2023-03-08 19:00:35,788 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392., pid=127, masterSystemTime=1678302035775 2023-03-08 19:00:35,789 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8., pid=132, masterSystemTime=1678302035778 2023-03-08 19:00:35,789 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28., pid=131, masterSystemTime=1678302035776 2023-03-08 19:00:35,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:35,790 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:35,790 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:35,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''} 2023-03-08 19:00:35,790 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:35,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:35,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:35,790 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302035790"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302035790"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302035790"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302035790"}]},"ts":"1678302035790"} 2023-03-08 19:00:35,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:35,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:35,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:35,791 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:35,791 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:35,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'} 2023-03-08 19:00:35,791 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:35,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:35,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:35,791 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302035791"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302035791"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302035791"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302035791"}]},"ts":"1678302035791"} 2023-03-08 19:00:35,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:35,791 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:35,791 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:35,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:35,792 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:35,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'} 2023-03-08 19:00:35,792 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:35,792 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302035792"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302035792"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302035792"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302035792"}]},"ts":"1678302035792"} 2023-03-08 19:00:35,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:35,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:35,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:35,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:35,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:35,793 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf 2023-03-08 19:00:35,794 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf 2023-03-08 19:00:35,794 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e3eaed3ac05b4e65948c1d4b5c75462d columnFamilyName cf 2023-03-08 19:00:35,794 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:35,794 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:35,795 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=123 2023-03-08 19:00:35,795 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=123, state=SUCCESS; OpenRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,46129,1678302018069 in 168 msec 2023-03-08 19:00:35,795 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(310): Store=e3eaed3ac05b4e65948c1d4b5c75462d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:35,795 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=121 2023-03-08 19:00:35,795 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=121, state=SUCCESS; OpenRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939 in 164 msec 2023-03-08 19:00:35,795 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf 2023-03-08 19:00:35,796 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf 2023-03-08 19:00:35,796 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=122 2023-03-08 19:00:35,796 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN in 330 msec 2023-03-08 19:00:35,796 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b8add49535b4b0366d051f3a48b2a333 columnFamilyName cf 2023-03-08 19:00:35,796 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf 2023-03-08 19:00:35,796 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf 2023-03-08 19:00:35,796 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=122, state=SUCCESS; OpenRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,41161,1678302018011 in 167 msec 2023-03-08 19:00:35,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:35,796 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ea43d1afb394fd7a05814d2275bdeffd columnFamilyName cf 2023-03-08 19:00:35,796 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN in 330 msec 2023-03-08 19:00:35,797 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(310): Store=b8add49535b4b0366d051f3a48b2a333/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:35,797 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN in 331 msec 2023-03-08 19:00:35,797 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(310): Store=ea43d1afb394fd7a05814d2275bdeffd/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:35,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:35,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:35,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:35,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:35,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:35,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:35,801 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened e3eaed3ac05b4e65948c1d4b5c75462d; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72535079, jitterRate=0.08085690438747406}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:35,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for e3eaed3ac05b4e65948c1d4b5c75462d: 2023-03-08 19:00:35,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:35,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:35,801 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d., pid=130, masterSystemTime=1678302035775 2023-03-08 19:00:35,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened b8add49535b4b0366d051f3a48b2a333; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67670880, jitterRate=0.008374691009521484}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:35,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened ea43d1afb394fd7a05814d2275bdeffd; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68040480, jitterRate=0.013882160186767578}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:35,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for ea43d1afb394fd7a05814d2275bdeffd: 2023-03-08 19:00:35,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for b8add49535b4b0366d051f3a48b2a333: 2023-03-08 19:00:35,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:35,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:35,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd., pid=128, masterSystemTime=1678302035776 2023-03-08 19:00:35,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333., pid=129, masterSystemTime=1678302035778 2023-03-08 19:00:35,803 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:35,803 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302035803"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302035803"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302035803"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302035803"}]},"ts":"1678302035803"} 2023-03-08 19:00:35,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:35,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:35,805 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:35,805 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302035805"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302035805"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302035805"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302035805"}]},"ts":"1678302035805"} 2023-03-08 19:00:35,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:35,805 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:35,806 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:35,806 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302035806"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302035806"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302035806"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302035806"}]},"ts":"1678302035806"} 2023-03-08 19:00:35,806 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=126 2023-03-08 19:00:35,806 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=126, state=SUCCESS; OpenRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,46129,1678302018069 in 178 msec 2023-03-08 19:00:35,807 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN in 341 msec 2023-03-08 19:00:35,808 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=128, resume processing ppid=125 2023-03-08 19:00:35,808 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=128, ppid=125, state=SUCCESS; OpenRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,41161,1678302018011 in 182 msec 2023-03-08 19:00:35,809 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=124 2023-03-08 19:00:35,809 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=124, state=SUCCESS; OpenRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,33875,1678302017939 in 182 msec 2023-03-08 19:00:35,809 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=125, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN in 343 msec 2023-03-08 19:00:35,811 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=124, resume processing ppid=120 2023-03-08 19:00:35,811 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=124, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN in 344 msec 2023-03-08 19:00:35,811 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302035811"}]},"ts":"1678302035811"} 2023-03-08 19:00:35,812 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=ENABLED in hbase:meta 2023-03-08 19:00:35,814 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1678302020890' was successfully enabled. 2023-03-08 19:00:35,815 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1678302020890 in 357 msec 2023-03-08 19:00:36,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-03-08 19:00:36,212 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1678302020890, procId: 120 completed 2023-03-08 19:00:36,213 DEBUG [Listener at localhost/45021] zookeeper.ReadOnlyZKClient(139): Connect 0x54f3b092 to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:36,218 DEBUG [Listener at localhost/45021] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bfe9d88, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:36,220 DEBUG [hconnection-0x6a24f0e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:36,223 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:57986, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:36,227 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-03-08 19:00:36,230 DEBUG [hconnection-0x6a24f0e6-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:36,231 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:55090, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:36,232 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-03-08 19:00:36,235 DEBUG [hconnection-0x6a24f0e6-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:36,239 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47288, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:36,240 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-03-08 19:00:36,244 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-03-08 19:00:36,247 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-03-08 19:00:36,256 INFO [Listener at localhost/45021] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,257 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$11(2418): Client=jenkins//172.31.2.20 disable testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-03-08 19:00:36,261 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302036261"}]},"ts":"1678302036261"} 2023-03-08 19:00:36,262 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=DISABLING in hbase:meta 2023-03-08 19:00:36,265 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1678302020890 to state=DISABLING 2023-03-08 19:00:36,266 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN}, {pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN}, {pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN}, {pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN}, {pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN}, {pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN}] 2023-03-08 19:00:36,268 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN 2023-03-08 19:00:36,268 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN 2023-03-08 19:00:36,268 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN 2023-03-08 19:00:36,268 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN 2023-03-08 19:00:36,268 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN 2023-03-08 19:00:36,269 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:36,269 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302036268"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302036268"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302036268"}]},"ts":"1678302036268"} 2023-03-08 19:00:36,269 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:36,269 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:36,269 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302036269"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302036269"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302036269"}]},"ts":"1678302036269"} 2023-03-08 19:00:36,269 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:36,269 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:36,270 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302036269"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302036269"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302036269"}]},"ts":"1678302036269"} 2023-03-08 19:00:36,270 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302036269"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302036269"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302036269"}]},"ts":"1678302036269"} 2023-03-08 19:00:36,269 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302036269"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302036269"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302036269"}]},"ts":"1678302036269"} 2023-03-08 19:00:36,271 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=140, ppid=138, state=RUNNABLE; CloseRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:36,271 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=141, ppid=139, state=RUNNABLE; CloseRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:36,272 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=135, state=RUNNABLE; CloseRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:36,273 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=137, state=RUNNABLE; CloseRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:36,274 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=144, ppid=136, state=RUNNABLE; CloseRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:36,275 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN 2023-03-08 19:00:36,276 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:36,276 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302036276"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302036276"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302036276"}]},"ts":"1678302036276"} 2023-03-08 19:00:36,278 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=134, state=RUNNABLE; CloseRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:36,424 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:36,425 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing ea43d1afb394fd7a05814d2275bdeffd, disabling compactions & flushes 2023-03-08 19:00:36,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:36,425 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:36,425 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. after waiting 0 ms 2023-03-08 19:00:36,425 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:36,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:36,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:36,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing b8add49535b4b0366d051f3a48b2a333, disabling compactions & flushes 2023-03-08 19:00:36,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 7cdf03195577655a4e5d1e7f4ff92392, disabling compactions & flushes 2023-03-08 19:00:36,427 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:36,427 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:36,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:36,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:36,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. after waiting 0 ms 2023-03-08 19:00:36,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. after waiting 0 ms 2023-03-08 19:00:36,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:36,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:36,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-03-08 19:00:36,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-03-08 19:00:36,431 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-03-08 19:00:36,431 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:36,431 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for ea43d1afb394fd7a05814d2275bdeffd: 2023-03-08 19:00:36,431 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:36,431 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 7cdf03195577655a4e5d1e7f4ff92392: 2023-03-08 19:00:36,431 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:36,431 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for b8add49535b4b0366d051f3a48b2a333: 2023-03-08 19:00:36,432 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:36,432 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:36,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing be16768f5782c50d4c94d5fdc74d3f28, disabling compactions & flushes 2023-03-08 19:00:36,433 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:36,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:36,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. after waiting 0 ms 2023-03-08 19:00:36,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:36,433 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=CLOSED 2023-03-08 19:00:36,433 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302036433"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302036433"}]},"ts":"1678302036433"} 2023-03-08 19:00:36,433 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:36,434 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:36,434 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing e3eaed3ac05b4e65948c1d4b5c75462d, disabling compactions & flushes 2023-03-08 19:00:36,434 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:36,434 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:36,434 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. after waiting 0 ms 2023-03-08 19:00:36,434 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:36,435 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=CLOSED 2023-03-08 19:00:36,435 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302036435"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302036435"}]},"ts":"1678302036435"} 2023-03-08 19:00:36,435 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:36,435 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:36,436 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 668bd868cc8589cf26791b7dda1212d8, disabling compactions & flushes 2023-03-08 19:00:36,436 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:36,436 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:36,436 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. after waiting 0 ms 2023-03-08 19:00:36,436 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:36,437 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=CLOSED 2023-03-08 19:00:36,437 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302036437"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302036437"}]},"ts":"1678302036437"} 2023-03-08 19:00:36,438 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-03-08 19:00:36,440 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=140, resume processing ppid=138 2023-03-08 19:00:36,440 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=144, resume processing ppid=136 2023-03-08 19:00:36,440 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=140, ppid=138, state=SUCCESS; CloseRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,41161,1678302018011 in 165 msec 2023-03-08 19:00:36,440 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=144, ppid=136, state=SUCCESS; CloseRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,46129,1678302018069 in 164 msec 2023-03-08 19:00:36,440 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=137 2023-03-08 19:00:36,440 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=137, state=SUCCESS; CloseRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,33875,1678302017939 in 166 msec 2023-03-08 19:00:36,441 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN in 174 msec 2023-03-08 19:00:36,441 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN in 174 msec 2023-03-08 19:00:36,441 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN in 174 msec 2023-03-08 19:00:36,443 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:36,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for be16768f5782c50d4c94d5fdc74d3f28: 2023-03-08 19:00:36,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-03-08 19:00:36,444 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:36,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for e3eaed3ac05b4e65948c1d4b5c75462d: 2023-03-08 19:00:36,444 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:36,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-03-08 19:00:36,445 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=CLOSED 2023-03-08 19:00:36,445 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302036445"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302036445"}]},"ts":"1678302036445"} 2023-03-08 19:00:36,445 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:36,445 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 668bd868cc8589cf26791b7dda1212d8: 2023-03-08 19:00:36,446 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:36,446 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=CLOSED 2023-03-08 19:00:36,446 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302036446"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302036446"}]},"ts":"1678302036446"} 2023-03-08 19:00:36,446 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:36,447 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=CLOSED 2023-03-08 19:00:36,447 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302036447"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302036447"}]},"ts":"1678302036447"} 2023-03-08 19:00:36,448 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=135 2023-03-08 19:00:36,448 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=135, state=SUCCESS; CloseRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,41161,1678302018011 in 174 msec 2023-03-08 19:00:36,449 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN in 182 msec 2023-03-08 19:00:36,449 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=141, resume processing ppid=139 2023-03-08 19:00:36,450 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=141, ppid=139, state=SUCCESS; CloseRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,46129,1678302018069 in 177 msec 2023-03-08 19:00:36,451 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=134 2023-03-08 19:00:36,451 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN in 183 msec 2023-03-08 19:00:36,451 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=134, state=SUCCESS; CloseRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939 in 172 msec 2023-03-08 19:00:36,452 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=134, resume processing ppid=133 2023-03-08 19:00:36,452 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN in 185 msec 2023-03-08 19:00:36,453 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302036453"}]},"ts":"1678302036453"} 2023-03-08 19:00:36,454 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=DISABLED in hbase:meta 2023-03-08 19:00:36,456 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1678302020890 to state=DISABLED 2023-03-08 19:00:36,457 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1678302020890 in 199 msec 2023-03-08 19:00:36,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-03-08 19:00:36,512 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1678302020890, procId: 133 completed 2023-03-08 19:00:36,521 INFO [Listener at localhost/45021] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb1-1678302020890-1678302036521 2023-03-08 19:00:36,522 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1580): Client=jenkins//172.31.2.20 snapshot request for:{ ss=hbase-failsafe-snaptb1-1678302020890-1678302036521 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:36,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1678302036522 (current time:1678302036522). 2023-03-08 19:00:36,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-03-08 19:00:36,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] zookeeper.ReadOnlyZKClient(139): Connect 0x4d33d870 to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:36,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d277b56, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:36,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:36,530 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:57990, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:36,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d33d870 to 127.0.0.1:59169 2023-03-08 19:00:36,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:36,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-03-08 19:00:36,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb1-1678302020890-1678302036521 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:36,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:36,533 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:36,534 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE; timeout=600000, timestamp=1678302636534 2023-03-08 19:00:36,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb1-1678302020890-1678302036521 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } 2023-03-08 19:00:36,534 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb1-1678302020890-1678302036521 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,535 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb1-1678302020890-1678302036521 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-03-08 19:00:36,535 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-03-08 19:00:36,543 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1678302020890-1678302036521 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } 2023-03-08 19:00:36,544 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:36,544 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:36,545 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:36,545 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:36,545 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:36,545 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:36,545 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:36,545 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:36,545 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:36,545 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:36,545 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:36,545 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:36,545 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:36,545 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:36,546 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:36,546 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:36,547 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:36,547 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:36,547 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:36,547 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:36,548 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:36,571 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb1-1678302020890-1678302036521 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } as finished. 2023-03-08 19:00:36,571 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb1-1678302020890-1678302036521 2023-03-08 19:00:36,571 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1678302020890-1678302036521 2023-03-08 19:00:36,590 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1678302020890-1678302036521 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/hbase-failsafe-snaptb1-1678302020890-1678302036521 2023-03-08 19:00:36,607 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb1-1678302020890-1678302036521 of table testRestoreSnapshot-1678302020890 completed 2023-03-08 19:00:36,607 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1678302020890-1678302036521 2023-03-08 19:00:36,607 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1678302020890-1678302036521 2023-03-08 19:00:36,607 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb1-1678302020890-1678302036521 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1678302020890 at 1678302036534Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1678302020890-1678302036521 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } at 1678302036543 (+9 ms)Consolidate snapshot: hbase-failsafe-snaptb1-1678302020890-1678302036521 at 1678302036571 (+28 ms)Loading Region manifests for hbase-failsafe-snaptb1-1678302020890-1678302036521 at 1678302036571Writing data manifest for hbase-failsafe-snaptb1-1678302020890-1678302036521 at 1678302036574 (+3 ms)Verifying snapshot: hbase-failsafe-snaptb1-1678302020890-1678302036521 at 1678302036582 (+8 ms)Snapshot hbase-failsafe-snaptb1-1678302020890-1678302036521 of table testRestoreSnapshot-1678302020890 completed at 1678302036607 (+25 ms) 2023-03-08 19:00:36,609 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE 2023-03-08 19:00:36,610 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1678302020890, type=EXCLUSIVE in 77 msec 2023-03-08 19:00:36,785 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-03-08 19:00:36,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1678302020890-1678302036521 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 } is done 2023-03-08 19:00:36,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb1-1678302020890-1678302036521 table=testRestoreSnapshot-1678302020890 type=FLUSH ttl=0 }' has completed, notifying client. 2023-03-08 19:00:36,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1678302020890 snapshot=name: "snaptb1-1678302020890" table: "testRestoreSnapshot-1678302020890" creation_time: 1678302030666 type: DISABLED version: 2 ttl: 0 ) 2023-03-08 19:00:36,801 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(880): Restore snapshot=snaptb1-1678302020890 as table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-03-08 19:00:36,812 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/.tabledesc/.tableinfo.0000000004 2023-03-08 19:00:36,813 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/.tabledesc/.tableinfo.0000000003 2023-03-08 19:00:36,813 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/.tabledesc/.tableinfo.0000000004 2023-03-08 19:00:36,814 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb1-1678302020890 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } 2023-03-08 19:00:36,816 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1678302020890" table: "testRestoreSnapshot-1678302020890" creation_time: 1678302030666 type: DISABLED version: 2 ttl: 0 2023-03-08 19:00:36,816 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,822 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,823 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 of snapshot=snaptb1-1678302020890 to region=f53d35b3eecf0fe701345a9b83dc29c9 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,826 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 of snapshot=snaptb1-1678302020890 to region=f53d35b3eecf0fe701345a9b83dc29c9 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,829 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 of snapshot=snaptb1-1678302020890 to region=f53d35b3eecf0fe701345a9b83dc29c9 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,831 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 of snapshot=snaptb1-1678302020890 to region=f53d35b3eecf0fe701345a9b83dc29c9 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,834 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 of snapshot=snaptb1-1678302020890 to region=f53d35b3eecf0fe701345a9b83dc29c9 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,837 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc of snapshot=snaptb1-1678302020890 to region=f53d35b3eecf0fe701345a9b83dc29c9 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,840 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d of snapshot=snaptb1-1678302020890 to region=f53d35b3eecf0fe701345a9b83dc29c9 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,843 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 of snapshot=snaptb1-1678302020890 to region=f53d35b3eecf0fe701345a9b83dc29c9 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,846 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 of snapshot=snaptb1-1678302020890 to region=f53d35b3eecf0fe701345a9b83dc29c9 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,849 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 of snapshot=snaptb1-1678302020890 to region=f53d35b3eecf0fe701345a9b83dc29c9 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,852 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 of snapshot=snaptb1-1678302020890 to region=f53d35b3eecf0fe701345a9b83dc29c9 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,855 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf of snapshot=snaptb1-1678302020890 to region=f53d35b3eecf0fe701345a9b83dc29c9 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,859 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:36,859 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:36,859 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:36,859 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:36,859 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:36,859 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:36,863 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 2a0e9bd776cc4f468bd20a74e4f61006 of snapshot=snaptb1-1678302020890 to region=668bd868cc8589cf26791b7dda1212d8 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,866 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 172aadf8677145f0b28293dbaeda5b9f of snapshot=snaptb1-1678302020890 to region=b8add49535b4b0366d051f3a48b2a333 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,866 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 657092e48c0b4ad0ab0fa0681948dafb of snapshot=snaptb1-1678302020890 to region=7cdf03195577655a4e5d1e7f4ff92392 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,867 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 4943e5745d97426d8d62901400379e84 of snapshot=snaptb1-1678302020890 to region=be16768f5782c50d4c94d5fdc74d3f28 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,867 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 7e72d8ad17d24d9f8bfdc8e8a32a99ed of snapshot=snaptb1-1678302020890 to region=ea43d1afb394fd7a05814d2275bdeffd table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,870 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 64a5b55bb32e4bff96aba52c33665694 of snapshot=snaptb1-1678302020890 to region=e3eaed3ac05b4e65948c1d4b5c75462d table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,878 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 5b72a399cd494fa4836e4a3e20c6b1ba of snapshot=snaptb1-1678302020890 to region=668bd868cc8589cf26791b7dda1212d8 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,879 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 57dba886df0248f48f93263d56e12c12 of snapshot=snaptb1-1678302020890 to region=be16768f5782c50d4c94d5fdc74d3f28 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,879 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink bd07ca0c4ad54ba0bb89f6a8507d4e9b of snapshot=snaptb1-1678302020890 to region=ea43d1afb394fd7a05814d2275bdeffd table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,883 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 9549786755914507847487aebb599d51 of snapshot=snaptb1-1678302020890 to region=b8add49535b4b0366d051f3a48b2a333 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,883 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink f24cdc04e3d949cf8fcfbd4389e9cdfb of snapshot=snaptb1-1678302020890 to region=7cdf03195577655a4e5d1e7f4ff92392 table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,885 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink bc3874f6411e445bb582265e908e860e of snapshot=snaptb1-1678302020890 to region=e3eaed3ac05b4e65948c1d4b5c75462d table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:36,894 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1678302020890" table: "testRestoreSnapshot-1678302020890" creation_time: 1678302030666 type: DISABLED version: 2 ttl: 0 2023-03-08 19:00:36,895 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302036895"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:36,895 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302036895"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:36,895 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302036895"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:36,895 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302036895"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:36,895 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302036895"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:36,895 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302036895"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:36,897 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-03-08 19:00:36,897 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'}] 2023-03-08 19:00:36,897 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302036896"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302036896"}]},"ts":"1678302036896"} 2023-03-08 19:00:36,897 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302036896"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302036896"}]},"ts":"1678302036896"} 2023-03-08 19:00:36,897 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302036896"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302036896"}]},"ts":"1678302036896"} 2023-03-08 19:00:36,897 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302036896"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302036896"}]},"ts":"1678302036896"} 2023-03-08 19:00:36,897 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302036896"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302036896"}]},"ts":"1678302036896"} 2023-03-08 19:00:36,897 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302036896"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302036896"}]},"ts":"1678302036896"} 2023-03-08 19:00:36,899 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-03-08 19:00:36,899 INFO [PEWorker-2] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-03-08 19:00:36,899 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'}] 2023-03-08 19:00:36,899 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb1-1678302020890 table=testRestoreSnapshot-1678302020890 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1678302020890 completed! 2023-03-08 19:00:36,902 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1678302020890 snapshot=name: "snaptb1-1678302020890" table: "testRestoreSnapshot-1678302020890" creation_time: 1678302030666 type: DISABLED version: 2 ttl: 0 ) in 102 msec 2023-03-08 19:00:37,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-03-08 19:00:37,054 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1678302020890, procId: 147 completed 2023-03-08 19:00:37,054 INFO [Listener at localhost/45021] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb1-1678302020890-1678302036521 2023-03-08 19:00:37,055 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(748): Client=jenkins//172.31.2.20 delete name: "hbase-failsafe-snaptb1-1678302020890-1678302036521" 2023-03-08 19:00:37,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb1-1678302020890-1678302036521 2023-03-08 19:00:37,057 INFO [Listener at localhost/45021] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1678302020890 2023-03-08 19:00:37,058 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$10(2385): Client=jenkins//172.31.2.20 enable testRestoreSnapshot-1678302020890 2023-03-08 19:00:37,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:37,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-03-08 19:00:37,061 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1678302020890 2023-03-08 19:00:37,061 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302037061"}]},"ts":"1678302037061"} 2023-03-08 19:00:37,062 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=ENABLING in hbase:meta 2023-03-08 19:00:37,065 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-03-08 19:00:37,065 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN}, {pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN}, {pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN}, {pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN}, {pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN}, {pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN}] 2023-03-08 19:00:37,066 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN 2023-03-08 19:00:37,067 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN 2023-03-08 19:00:37,067 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN 2023-03-08 19:00:37,067 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN 2023-03-08 19:00:37,067 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:37,068 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:37,068 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:37,068 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:37,068 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN 2023-03-08 19:00:37,068 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN 2023-03-08 19:00:37,069 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:37,069 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-03-08 19:00:37,218 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase6.apache.org=0} racks are {/default-rack=0} 2023-03-08 19:00:37,218 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-08 19:00:37,218 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-03-08 19:00:37,218 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-03-08 19:00:37,218 DEBUG [jenkins-hbase6:43879] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-03-08 19:00:37,221 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:37,221 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:37,221 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302037221"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302037221"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302037221"}]},"ts":"1678302037221"} 2023-03-08 19:00:37,221 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:37,221 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:37,221 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302037221"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302037221"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302037221"}]},"ts":"1678302037221"} 2023-03-08 19:00:37,222 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302037221"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302037221"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302037221"}]},"ts":"1678302037221"} 2023-03-08 19:00:37,221 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:37,221 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302037221"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302037221"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302037221"}]},"ts":"1678302037221"} 2023-03-08 19:00:37,222 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302037221"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302037221"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302037221"}]},"ts":"1678302037221"} 2023-03-08 19:00:37,223 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=151, state=RUNNABLE; OpenRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:37,224 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=150, state=RUNNABLE; OpenRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:37,224 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=157, ppid=153, state=RUNNABLE; OpenRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:37,225 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=158, ppid=152, state=RUNNABLE; OpenRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:37,226 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=159, ppid=154, state=RUNNABLE; OpenRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:37,226 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:37,226 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302037226"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302037226"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302037226"}]},"ts":"1678302037226"} 2023-03-08 19:00:37,228 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=160, ppid=149, state=RUNNABLE; OpenRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:37,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-03-08 19:00:37,377 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:37,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''} 2023-03-08 19:00:37,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:37,377 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:37,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:37,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'} 2023-03-08 19:00:37,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:37,377 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:37,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:37,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'} 2023-03-08 19:00:37,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:37,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:37,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:37,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:37,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:37,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:37,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:37,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:37,379 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:37,379 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:37,379 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:37,379 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf 2023-03-08 19:00:37,380 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf 2023-03-08 19:00:37,380 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf 2023-03-08 19:00:37,380 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e3eaed3ac05b4e65948c1d4b5c75462d columnFamilyName cf 2023-03-08 19:00:37,380 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf 2023-03-08 19:00:37,380 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf 2023-03-08 19:00:37,380 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf 2023-03-08 19:00:37,380 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 668bd868cc8589cf26791b7dda1212d8 columnFamilyName cf 2023-03-08 19:00:37,380 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region be16768f5782c50d4c94d5fdc74d3f28 columnFamilyName cf 2023-03-08 19:00:37,396 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-2a0e9bd776cc4f468bd20a74e4f61006 2023-03-08 19:00:37,396 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-4943e5745d97426d8d62901400379e84 2023-03-08 19:00:37,396 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-64a5b55bb32e4bff96aba52c33665694 2023-03-08 19:00:37,404 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-57dba886df0248f48f93263d56e12c12 2023-03-08 19:00:37,405 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-bc3874f6411e445bb582265e908e860e 2023-03-08 19:00:37,405 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(310): Store=be16768f5782c50d4c94d5fdc74d3f28/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:37,405 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(310): Store=e3eaed3ac05b4e65948c1d4b5c75462d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:37,406 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-5b72a399cd494fa4836e4a3e20c6b1ba 2023-03-08 19:00:37,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:37,406 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(310): Store=668bd868cc8589cf26791b7dda1212d8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:37,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:37,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:37,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:37,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:37,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:37,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:37,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:37,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened be16768f5782c50d4c94d5fdc74d3f28; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74831734, jitterRate=0.11507973074913025}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:37,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for be16768f5782c50d4c94d5fdc74d3f28: 2023-03-08 19:00:37,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:37,411 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened e3eaed3ac05b4e65948c1d4b5c75462d; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73128275, jitterRate=0.08969621360301971}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:37,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for e3eaed3ac05b4e65948c1d4b5c75462d: 2023-03-08 19:00:37,411 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28., pid=156, masterSystemTime=1678302037376 2023-03-08 19:00:37,411 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 668bd868cc8589cf26791b7dda1212d8; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65942128, jitterRate=-0.01738572120666504}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:37,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 668bd868cc8589cf26791b7dda1212d8: 2023-03-08 19:00:37,412 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d., pid=159, masterSystemTime=1678302037374 2023-03-08 19:00:37,412 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8., pid=160, masterSystemTime=1678302037376 2023-03-08 19:00:37,413 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:37,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:37,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:37,413 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'} 2023-03-08 19:00:37,413 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:37,413 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:37,413 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:37,413 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:37,413 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302037413"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302037413"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302037413"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302037413"}]},"ts":"1678302037413"} 2023-03-08 19:00:37,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:37,414 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:37,413 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:37,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'} 2023-03-08 19:00:37,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:37,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:37,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:37,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:37,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:37,414 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:37,415 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302037414"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302037414"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302037414"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302037414"}]},"ts":"1678302037414"} 2023-03-08 19:00:37,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:37,415 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:37,415 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:37,415 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:37,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'} 2023-03-08 19:00:37,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:37,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:37,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:37,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:37,416 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:37,416 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302037416"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302037416"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302037416"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302037416"}]},"ts":"1678302037416"} 2023-03-08 19:00:37,417 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:37,418 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf 2023-03-08 19:00:37,418 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf 2023-03-08 19:00:37,419 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=150 2023-03-08 19:00:37,419 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=150, state=SUCCESS; OpenRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,41161,1678302018011 in 193 msec 2023-03-08 19:00:37,419 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ea43d1afb394fd7a05814d2275bdeffd columnFamilyName cf 2023-03-08 19:00:37,419 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf 2023-03-08 19:00:37,419 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf 2023-03-08 19:00:37,419 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=159, resume processing ppid=154 2023-03-08 19:00:37,419 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=159, ppid=154, state=SUCCESS; OpenRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,46129,1678302018069 in 191 msec 2023-03-08 19:00:37,420 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7cdf03195577655a4e5d1e7f4ff92392 columnFamilyName cf 2023-03-08 19:00:37,420 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=160, resume processing ppid=149 2023-03-08 19:00:37,420 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN in 354 msec 2023-03-08 19:00:37,420 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=160, ppid=149, state=SUCCESS; OpenRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939 in 190 msec 2023-03-08 19:00:37,421 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN in 354 msec 2023-03-08 19:00:37,421 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN in 355 msec 2023-03-08 19:00:37,424 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:37,427 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf 2023-03-08 19:00:37,427 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf 2023-03-08 19:00:37,427 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b8add49535b4b0366d051f3a48b2a333 columnFamilyName cf 2023-03-08 19:00:37,430 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1678302020890' 2023-03-08 19:00:37,434 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-7e72d8ad17d24d9f8bfdc8e8a32a99ed 2023-03-08 19:00:37,434 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-657092e48c0b4ad0ab0fa0681948dafb 2023-03-08 19:00:37,441 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-172aadf8677145f0b28293dbaeda5b9f 2023-03-08 19:00:37,444 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-bd07ca0c4ad54ba0bb89f6a8507d4e9b 2023-03-08 19:00:37,445 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(310): Store=ea43d1afb394fd7a05814d2275bdeffd/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:37,446 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-f24cdc04e3d949cf8fcfbd4389e9cdfb 2023-03-08 19:00:37,446 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(310): Store=7cdf03195577655a4e5d1e7f4ff92392/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:37,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:37,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:37,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:37,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:37,449 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-9549786755914507847487aebb599d51 2023-03-08 19:00:37,449 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(310): Store=b8add49535b4b0366d051f3a48b2a333/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:37,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:37,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:37,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:37,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:37,452 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened ea43d1afb394fd7a05814d2275bdeffd; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70856607, jitterRate=0.0558457225561142}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:37,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for ea43d1afb394fd7a05814d2275bdeffd: 2023-03-08 19:00:37,452 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 7cdf03195577655a4e5d1e7f4ff92392; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61172119, jitterRate=-0.0884643942117691}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:37,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 7cdf03195577655a4e5d1e7f4ff92392: 2023-03-08 19:00:37,452 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd., pid=157, masterSystemTime=1678302037376 2023-03-08 19:00:37,453 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392., pid=155, masterSystemTime=1678302037374 2023-03-08 19:00:37,454 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:37,455 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:37,455 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302037455"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302037455"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302037455"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302037455"}]},"ts":"1678302037455"} 2023-03-08 19:00:37,455 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:37,455 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:37,455 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:37,455 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:37,455 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:37,456 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302037455"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302037455"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302037455"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302037455"}]},"ts":"1678302037455"} 2023-03-08 19:00:37,456 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened b8add49535b4b0366d051f3a48b2a333; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63086631, jitterRate=-0.05993594229221344}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:37,457 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for b8add49535b4b0366d051f3a48b2a333: 2023-03-08 19:00:37,457 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333., pid=158, masterSystemTime=1678302037376 2023-03-08 19:00:37,458 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:37,459 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:37,459 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=157, resume processing ppid=153 2023-03-08 19:00:37,459 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=157, ppid=153, state=SUCCESS; OpenRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,41161,1678302018011 in 232 msec 2023-03-08 19:00:37,459 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=151 2023-03-08 19:00:37,459 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:37,460 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=151, state=SUCCESS; OpenRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,46129,1678302018069 in 234 msec 2023-03-08 19:00:37,460 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302037459"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302037459"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302037459"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302037459"}]},"ts":"1678302037459"} 2023-03-08 19:00:37,460 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN in 394 msec 2023-03-08 19:00:37,461 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN in 395 msec 2023-03-08 19:00:37,463 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=158, resume processing ppid=152 2023-03-08 19:00:37,463 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=158, ppid=152, state=SUCCESS; OpenRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,33875,1678302017939 in 236 msec 2023-03-08 19:00:37,464 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=152, resume processing ppid=148 2023-03-08 19:00:37,464 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN in 398 msec 2023-03-08 19:00:37,465 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302037465"}]},"ts":"1678302037465"} 2023-03-08 19:00:37,466 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=ENABLED in hbase:meta 2023-03-08 19:00:37,467 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1678302020890' was successfully enabled. 2023-03-08 19:00:37,469 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1678302020890 in 410 msec 2023-03-08 19:00:37,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-03-08 19:00:37,812 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1678302020890, procId: 148 completed 2023-03-08 19:00:37,813 DEBUG [Listener at localhost/45021] zookeeper.ReadOnlyZKClient(139): Connect 0x3a5b8d75 to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:37,818 DEBUG [Listener at localhost/45021] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@131d42f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:37,820 DEBUG [hconnection-0x29616110-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:37,822 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:57992, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:37,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:37,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,021 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-03-08 19:00:38,025 DEBUG [hconnection-0x29616110-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:38,030 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:55104, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:38,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,278 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-03-08 19:00:38,281 DEBUG [hconnection-0x29616110-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:38,286 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47294, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:38,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,534 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-03-08 19:00:38,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,589 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29e7c4cf to 127.0.0.1:59169 2023-03-08 19:00:38,589 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:38,592 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54f3b092 to 127.0.0.1:59169 2023-03-08 19:00:38,592 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:38,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,763 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-03-08 19:00:38,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:38,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,001 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-03-08 19:00:39,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:39,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:40,852 INFO [Listener at localhost/45021] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1678302020890 2023-03-08 19:00:40,852 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$11(2418): Client=jenkins//172.31.2.20 disable testRestoreSnapshot-1678302020890 2023-03-08 19:00:40,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:40,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-03-08 19:00:40,856 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302040856"}]},"ts":"1678302040856"} 2023-03-08 19:00:40,858 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=DISABLING in hbase:meta 2023-03-08 19:00:40,860 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1678302020890 to state=DISABLING 2023-03-08 19:00:40,860 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN}, {pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN}, {pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN}, {pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN}, {pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN}, {pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN}] 2023-03-08 19:00:40,862 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN 2023-03-08 19:00:40,862 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN 2023-03-08 19:00:40,862 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN 2023-03-08 19:00:40,862 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN 2023-03-08 19:00:40,862 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN 2023-03-08 19:00:40,863 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:40,863 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:40,863 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:40,864 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:40,863 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:40,864 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302040864"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302040864"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302040864"}]},"ts":"1678302040864"} 2023-03-08 19:00:40,864 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302040863"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302040863"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302040863"}]},"ts":"1678302040863"} 2023-03-08 19:00:40,864 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302040863"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302040863"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302040863"}]},"ts":"1678302040863"} 2023-03-08 19:00:40,863 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302040863"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302040863"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302040863"}]},"ts":"1678302040863"} 2023-03-08 19:00:40,863 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302040863"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302040863"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302040863"}]},"ts":"1678302040863"} 2023-03-08 19:00:40,865 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=168, ppid=166, state=RUNNABLE; CloseRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:40,866 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=169, ppid=167, state=RUNNABLE; CloseRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:40,867 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=170, ppid=162, state=RUNNABLE; CloseRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:40,868 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=171, ppid=164, state=RUNNABLE; CloseRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:40,868 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN 2023-03-08 19:00:40,868 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=172, ppid=163, state=RUNNABLE; CloseRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:40,869 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:40,869 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302040869"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302040869"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302040869"}]},"ts":"1678302040869"} 2023-03-08 19:00:40,871 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=173, ppid=165, state=RUNNABLE; CloseRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:41,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:41,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:41,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing e3eaed3ac05b4e65948c1d4b5c75462d, disabling compactions & flushes 2023-03-08 19:00:41,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing ea43d1afb394fd7a05814d2275bdeffd, disabling compactions & flushes 2023-03-08 19:00:41,018 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:41,018 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:41,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:41,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:41,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. after waiting 0 ms 2023-03-08 19:00:41,019 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:41,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. after waiting 0 ms 2023-03-08 19:00:41,019 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:41,019 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:41,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 668bd868cc8589cf26791b7dda1212d8, disabling compactions & flushes 2023-03-08 19:00:41,020 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:41,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:41,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. after waiting 0 ms 2023-03-08 19:00:41,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:41,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-03-08 19:00:41,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-03-08 19:00:41,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-03-08 19:00:41,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:41,031 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 668bd868cc8589cf26791b7dda1212d8: 2023-03-08 19:00:41,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:41,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:41,031 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for e3eaed3ac05b4e65948c1d4b5c75462d: 2023-03-08 19:00:41,031 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for ea43d1afb394fd7a05814d2275bdeffd: 2023-03-08 19:00:41,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:41,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:41,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing b8add49535b4b0366d051f3a48b2a333, disabling compactions & flushes 2023-03-08 19:00:41,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:41,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:41,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. after waiting 0 ms 2023-03-08 19:00:41,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:41,034 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=CLOSED 2023-03-08 19:00:41,034 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302041034"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302041034"}]},"ts":"1678302041034"} 2023-03-08 19:00:41,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:41,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:41,037 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=CLOSED 2023-03-08 19:00:41,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing be16768f5782c50d4c94d5fdc74d3f28, disabling compactions & flushes 2023-03-08 19:00:41,038 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:41,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:41,038 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302041037"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302041037"}]},"ts":"1678302041037"} 2023-03-08 19:00:41,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. after waiting 0 ms 2023-03-08 19:00:41,038 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:41,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:41,038 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:41,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 7cdf03195577655a4e5d1e7f4ff92392, disabling compactions & flushes 2023-03-08 19:00:41,039 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:41,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:41,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. after waiting 0 ms 2023-03-08 19:00:41,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:41,040 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=CLOSED 2023-03-08 19:00:41,040 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302041040"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302041040"}]},"ts":"1678302041040"} 2023-03-08 19:00:41,044 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=170, resume processing ppid=162 2023-03-08 19:00:41,045 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=170, ppid=162, state=SUCCESS; CloseRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939 in 171 msec 2023-03-08 19:00:41,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-03-08 19:00:41,050 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=168, resume processing ppid=166 2023-03-08 19:00:41,050 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=168, ppid=166, state=SUCCESS; CloseRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,41161,1678302018011 in 175 msec 2023-03-08 19:00:41,050 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=162, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN in 185 msec 2023-03-08 19:00:41,051 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=169, resume processing ppid=167 2023-03-08 19:00:41,051 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:41,051 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=169, ppid=167, state=SUCCESS; CloseRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,46129,1678302018069 in 179 msec 2023-03-08 19:00:41,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for b8add49535b4b0366d051f3a48b2a333: 2023-03-08 19:00:41,052 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=166, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN in 190 msec 2023-03-08 19:00:41,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:41,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-03-08 19:00:41,053 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=167, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN in 191 msec 2023-03-08 19:00:41,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-03-08 19:00:41,053 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=CLOSED 2023-03-08 19:00:41,053 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302041053"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302041053"}]},"ts":"1678302041053"} 2023-03-08 19:00:41,054 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:41,054 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for be16768f5782c50d4c94d5fdc74d3f28: 2023-03-08 19:00:41,054 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:41,054 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 7cdf03195577655a4e5d1e7f4ff92392: 2023-03-08 19:00:41,056 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:41,057 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=CLOSED 2023-03-08 19:00:41,057 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:41,057 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302041056"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302041056"}]},"ts":"1678302041056"} 2023-03-08 19:00:41,058 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=173, resume processing ppid=165 2023-03-08 19:00:41,058 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=173, ppid=165, state=SUCCESS; CloseRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,33875,1678302017939 in 184 msec 2023-03-08 19:00:41,058 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=CLOSED 2023-03-08 19:00:41,058 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302041058"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302041058"}]},"ts":"1678302041058"} 2023-03-08 19:00:41,060 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN in 198 msec 2023-03-08 19:00:41,060 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=172, resume processing ppid=163 2023-03-08 19:00:41,060 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=172, ppid=163, state=SUCCESS; CloseRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,41161,1678302018011 in 190 msec 2023-03-08 19:00:41,061 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=171, resume processing ppid=164 2023-03-08 19:00:41,061 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=163, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN in 200 msec 2023-03-08 19:00:41,061 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=171, ppid=164, state=SUCCESS; CloseRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,46129,1678302018069 in 192 msec 2023-03-08 19:00:41,063 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=164, resume processing ppid=161 2023-03-08 19:00:41,063 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN in 201 msec 2023-03-08 19:00:41,064 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302041064"}]},"ts":"1678302041064"} 2023-03-08 19:00:41,065 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=DISABLED in hbase:meta 2023-03-08 19:00:41,067 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1678302020890 to state=DISABLED 2023-03-08 19:00:41,068 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1678302020890 in 214 msec 2023-03-08 19:00:41,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-03-08 19:00:41,107 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1678302020890, procId: 161 completed 2023-03-08 19:00:41,112 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$5(2228): Client=jenkins//172.31.2.20 delete testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,119 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,121 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=174, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-03-08 19:00:41,127 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:41,127 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:41,127 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:41,127 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:41,127 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:41,127 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:41,129 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/recovered.edits] 2023-03-08 19:00:41,129 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/recovered.edits] 2023-03-08 19:00:41,130 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/recovered.edits] 2023-03-08 19:00:41,130 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/recovered.edits] 2023-03-08 19:00:41,130 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/recovered.edits] 2023-03-08 19:00:41,130 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/recovered.edits] 2023-03-08 19:00:41,134 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-657092e48c0b4ad0ab0fa0681948dafb to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-657092e48c0b4ad0ab0fa0681948dafb 2023-03-08 19:00:41,134 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-64a5b55bb32e4bff96aba52c33665694 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-64a5b55bb32e4bff96aba52c33665694 2023-03-08 19:00:41,134 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-4943e5745d97426d8d62901400379e84 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-4943e5745d97426d8d62901400379e84 2023-03-08 19:00:41,134 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-2a0e9bd776cc4f468bd20a74e4f61006 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-2a0e9bd776cc4f468bd20a74e4f61006 2023-03-08 19:00:41,134 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-7e72d8ad17d24d9f8bfdc8e8a32a99ed to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-7e72d8ad17d24d9f8bfdc8e8a32a99ed 2023-03-08 19:00:41,159 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-172aadf8677145f0b28293dbaeda5b9f to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-172aadf8677145f0b28293dbaeda5b9f 2023-03-08 19:00:41,161 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a5b8d75 to 127.0.0.1:59169 2023-03-08 19:00:41,161 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:41,161 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-bc3874f6411e445bb582265e908e860e to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-bc3874f6411e445bb582265e908e860e 2023-03-08 19:00:41,162 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-5b72a399cd494fa4836e4a3e20c6b1ba to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-5b72a399cd494fa4836e4a3e20c6b1ba 2023-03-08 19:00:41,162 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-f24cdc04e3d949cf8fcfbd4389e9cdfb to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-f24cdc04e3d949cf8fcfbd4389e9cdfb 2023-03-08 19:00:41,162 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-bd07ca0c4ad54ba0bb89f6a8507d4e9b to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-bd07ca0c4ad54ba0bb89f6a8507d4e9b 2023-03-08 19:00:41,162 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-9549786755914507847487aebb599d51 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-9549786755914507847487aebb599d51 2023-03-08 19:00:41,162 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-57dba886df0248f48f93263d56e12c12 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-57dba886df0248f48f93263d56e12c12 2023-03-08 19:00:41,166 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/recovered.edits/27.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/recovered.edits/27.seqid 2023-03-08 19:00:41,166 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/recovered.edits/27.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/recovered.edits/27.seqid 2023-03-08 19:00:41,166 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/recovered.edits/27.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/recovered.edits/27.seqid 2023-03-08 19:00:41,166 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/recovered.edits/27.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/recovered.edits/27.seqid 2023-03-08 19:00:41,166 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:41,167 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:41,167 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:41,167 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/recovered.edits/27.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/recovered.edits/27.seqid 2023-03-08 19:00:41,167 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:41,167 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/recovered.edits/27.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/recovered.edits/27.seqid 2023-03-08 19:00:41,167 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:41,168 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:41,168 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1678302020890 regions 2023-03-08 19:00:41,168 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9 2023-03-08 19:00:41,169 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf] 2023-03-08 19:00:41,170 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 2023-03-08 19:00:41,171 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 2023-03-08 19:00:41,172 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 2023-03-08 19:00:41,173 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 2023-03-08 19:00:41,174 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 2023-03-08 19:00:41,175 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc 2023-03-08 19:00:41,175 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d 2023-03-08 19:00:41,176 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 2023-03-08 19:00:41,177 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 2023-03-08 19:00:41,178 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 2023-03-08 19:00:41,179 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 2023-03-08 19:00:41,180 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf 2023-03-08 19:00:41,181 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9 2023-03-08 19:00:41,182 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=174, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,203 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1678302020890 from hbase:meta 2023-03-08 19:00:41,205 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1678302020890' descriptor. 2023-03-08 19:00:41,206 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=174, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,206 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1678302020890' from region states. 2023-03-08 19:00:41,206 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302041206"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:41,206 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302041206"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:41,206 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302041206"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:41,206 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302041206"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:41,206 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302041206"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:41,206 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302041206"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:41,208 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-03-08 19:00:41,208 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''}] 2023-03-08 19:00:41,208 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1678302020890' as deleted. 2023-03-08 19:00:41,208 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1678302041208"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:41,210 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1678302020890 state from META 2023-03-08 19:00:41,213 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=174, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,214 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=174, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1678302020890 in 100 msec 2023-03-08 19:00:41,281 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-08 19:00:41,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-03-08 19:00:41,373 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1678302020890, procId: 174 completed 2023-03-08 19:00:41,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=175, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRestoreSnapshot-1678302020890 snapshot=name: "snaptb1-1678302020890" table: "testRestoreSnapshot-1678302020890" creation_time: 1678302030666 type: DISABLED version: 2 ttl: 0 ) 2023-03-08 19:00:41,398 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(750): Clone snapshot=snaptb1-1678302020890 as table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-03-08 19:00:41,411 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/.tabledesc/.tableinfo.0000000001 2023-03-08 19:00:41,413 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1678302020890" table: "testRestoreSnapshot-1678302020890" creation_time: 1678302030666 type: DISABLED version: 2 ttl: 0 2023-03-08 19:00:41,413 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,414 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,416 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,418 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,420 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,422 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,424 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,425 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,427 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,428 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,430 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,432 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,433 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,435 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:41,435 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:41,435 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:41,435 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:41,435 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:41,435 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:41,435 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=e3eaed3ac05b4e65948c1d4b5c75462d as e3eaed3ac05b4e65948c1d4b5c75462d in snapshot snaptb1-1678302020890 2023-03-08 19:00:41,435 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=be16768f5782c50d4c94d5fdc74d3f28 as be16768f5782c50d4c94d5fdc74d3f28 in snapshot snaptb1-1678302020890 2023-03-08 19:00:41,435 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=668bd868cc8589cf26791b7dda1212d8 as 668bd868cc8589cf26791b7dda1212d8 in snapshot snaptb1-1678302020890 2023-03-08 19:00:41,435 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=ea43d1afb394fd7a05814d2275bdeffd as ea43d1afb394fd7a05814d2275bdeffd in snapshot snaptb1-1678302020890 2023-03-08 19:00:41,435 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=b8add49535b4b0366d051f3a48b2a333 as b8add49535b4b0366d051f3a48b2a333 in snapshot snaptb1-1678302020890 2023-03-08 19:00:41,435 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=7cdf03195577655a4e5d1e7f4ff92392 as 7cdf03195577655a4e5d1e7f4ff92392 in snapshot snaptb1-1678302020890 2023-03-08 19:00:41,437 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(7675): creating {ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1678302020890', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:41,437 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(7675): creating {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1678302020890', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:41,437 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(7675): creating {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1678302020890', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:41,437 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(7675): creating {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1678302020890', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:41,437 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7675): creating {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1678302020890', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:41,437 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(7675): creating {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1678302020890', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:41,462 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:41,462 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 2a0e9bd776cc4f468bd20a74e4f61006 from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,467 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:41,467 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 64a5b55bb32e4bff96aba52c33665694 from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,467 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:41,467 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 172aadf8677145f0b28293dbaeda5b9f from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,472 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 9549786755914507847487aebb599d51 from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,472 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:41,472 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 7e72d8ad17d24d9f8bfdc8e8a32a99ed from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,474 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 5b72a399cd494fa4836e4a3e20c6b1ba from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,474 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink bc3874f6411e445bb582265e908e860e from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,475 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink bd07ca0c4ad54ba0bb89f6a8507d4e9b from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,476 WARN [DataStreamer for file /user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/.regioninfo] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-03-08 19:00:41,478 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1604): Closing b8add49535b4b0366d051f3a48b2a333, disabling compactions & flushes 2023-03-08 19:00:41,478 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:41,478 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:41,478 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. after waiting 0 ms 2023-03-08 19:00:41,478 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:41,478 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:41,478 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 4943e5745d97426d8d62901400379e84 from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,478 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:41,478 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1558): Region close journal for b8add49535b4b0366d051f3a48b2a333: 2023-03-08 19:00:41,479 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:41,479 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 657092e48c0b4ad0ab0fa0681948dafb from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,480 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1604): Closing 668bd868cc8589cf26791b7dda1212d8, disabling compactions & flushes 2023-03-08 19:00:41,480 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:41,480 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:41,480 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. after waiting 0 ms 2023-03-08 19:00:41,480 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:41,480 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:41,480 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1604): Closing e3eaed3ac05b4e65948c1d4b5c75462d, disabling compactions & flushes 2023-03-08 19:00:41,480 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1558): Region close journal for 668bd868cc8589cf26791b7dda1212d8: 2023-03-08 19:00:41,480 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:41,480 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:41,480 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. after waiting 0 ms 2023-03-08 19:00:41,481 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:41,481 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:41,481 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1558): Region close journal for e3eaed3ac05b4e65948c1d4b5c75462d: 2023-03-08 19:00:41,481 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1604): Closing ea43d1afb394fd7a05814d2275bdeffd, disabling compactions & flushes 2023-03-08 19:00:41,481 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:41,481 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:41,481 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. after waiting 0 ms 2023-03-08 19:00:41,481 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:41,481 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:41,481 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1558): Region close journal for ea43d1afb394fd7a05814d2275bdeffd: 2023-03-08 19:00:41,482 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 57dba886df0248f48f93263d56e12c12 from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,482 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink f24cdc04e3d949cf8fcfbd4389e9cdfb from cloned region in snapshot snaptb1-1678302020890 to table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:41,484 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1604): Closing be16768f5782c50d4c94d5fdc74d3f28, disabling compactions & flushes 2023-03-08 19:00:41,484 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:41,484 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:41,484 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. after waiting 0 ms 2023-03-08 19:00:41,484 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:41,484 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:41,484 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1558): Region close journal for be16768f5782c50d4c94d5fdc74d3f28: 2023-03-08 19:00:41,484 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1604): Closing 7cdf03195577655a4e5d1e7f4ff92392, disabling compactions & flushes 2023-03-08 19:00:41,484 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:41,484 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:41,484 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. after waiting 0 ms 2023-03-08 19:00:41,484 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:41,484 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:41,484 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1558): Region close journal for 7cdf03195577655a4e5d1e7f4ff92392: 2023-03-08 19:00:41,484 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1678302020890" table: "testRestoreSnapshot-1678302020890" creation_time: 1678302030666 type: DISABLED version: 2 ttl: 0 2023-03-08 19:00:41,485 INFO [PEWorker-5] procedure.CloneSnapshotProcedure$1(411): Clone snapshot=snaptb1-1678302020890 on table=testRestoreSnapshot-1678302020890 completed! 2023-03-08 19:00:41,489 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302041489"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302041489"}]},"ts":"1678302041489"} 2023-03-08 19:00:41,489 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302041489"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302041489"}]},"ts":"1678302041489"} 2023-03-08 19:00:41,489 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302041489"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302041489"}]},"ts":"1678302041489"} 2023-03-08 19:00:41,489 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302041489"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302041489"}]},"ts":"1678302041489"} 2023-03-08 19:00:41,489 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302041489"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302041489"}]},"ts":"1678302041489"} 2023-03-08 19:00:41,489 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302041489"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302041489"}]},"ts":"1678302041489"} 2023-03-08 19:00:41,491 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-03-08 19:00:41,492 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302041492"}]},"ts":"1678302041492"} 2023-03-08 19:00:41,493 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=ENABLING in hbase:meta 2023-03-08 19:00:41,498 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase6.apache.org=0} racks are {/default-rack=0} 2023-03-08 19:00:41,498 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-08 19:00:41,498 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-03-08 19:00:41,498 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-03-08 19:00:41,498 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-03-08 19:00:41,499 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN}, {pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN}, {pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN}, {pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN}, {pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN}, {pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN}] 2023-03-08 19:00:41,500 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN 2023-03-08 19:00:41,501 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN 2023-03-08 19:00:41,501 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN 2023-03-08 19:00:41,501 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN 2023-03-08 19:00:41,501 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN 2023-03-08 19:00:41,501 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,41161,1678302018011; forceNewPlan=false, retain=false 2023-03-08 19:00:41,501 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,46129,1678302018069; forceNewPlan=false, retain=false 2023-03-08 19:00:41,502 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,33875,1678302017939; forceNewPlan=false, retain=false 2023-03-08 19:00:41,502 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,46129,1678302018069; forceNewPlan=false, retain=false 2023-03-08 19:00:41,502 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,41161,1678302018011; forceNewPlan=false, retain=false 2023-03-08 19:00:41,502 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN 2023-03-08 19:00:41,503 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,33875,1678302017939; forceNewPlan=false, retain=false 2023-03-08 19:00:41,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-03-08 19:00:41,652 INFO [jenkins-hbase6:43879] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-03-08 19:00:41,655 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:41,655 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:41,655 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:41,655 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:41,655 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302041655"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302041655"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302041655"}]},"ts":"1678302041655"} 2023-03-08 19:00:41,655 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:41,655 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302041655"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302041655"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302041655"}]},"ts":"1678302041655"} 2023-03-08 19:00:41,655 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302041655"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302041655"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302041655"}]},"ts":"1678302041655"} 2023-03-08 19:00:41,655 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302041655"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302041655"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302041655"}]},"ts":"1678302041655"} 2023-03-08 19:00:41,656 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302041655"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302041655"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302041655"}]},"ts":"1678302041655"} 2023-03-08 19:00:41,657 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=182, ppid=179, state=RUNNABLE; OpenRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:41,659 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=183, ppid=181, state=RUNNABLE; OpenRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:41,660 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=184, ppid=177, state=RUNNABLE; OpenRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:41,660 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=185, ppid=180, state=RUNNABLE; OpenRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:41,661 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:41,661 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=186, ppid=178, state=RUNNABLE; OpenRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:41,661 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302041661"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302041661"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302041661"}]},"ts":"1678302041661"} 2023-03-08 19:00:41,663 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=187, ppid=176, state=RUNNABLE; OpenRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:41,813 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:41,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'} 2023-03-08 19:00:41,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:41,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:41,814 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:41,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:41,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:41,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'} 2023-03-08 19:00:41,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:41,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:41,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:41,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:41,815 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:41,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'} 2023-03-08 19:00:41,815 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:41,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:41,816 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:41,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:41,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:41,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:41,816 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf 2023-03-08 19:00:41,816 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf 2023-03-08 19:00:41,816 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf 2023-03-08 19:00:41,816 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf 2023-03-08 19:00:41,817 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:41,817 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ea43d1afb394fd7a05814d2275bdeffd columnFamilyName cf 2023-03-08 19:00:41,817 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region be16768f5782c50d4c94d5fdc74d3f28 columnFamilyName cf 2023-03-08 19:00:41,818 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf 2023-03-08 19:00:41,818 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf 2023-03-08 19:00:41,818 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 668bd868cc8589cf26791b7dda1212d8 columnFamilyName cf 2023-03-08 19:00:41,826 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-7e72d8ad17d24d9f8bfdc8e8a32a99ed 2023-03-08 19:00:41,826 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-4943e5745d97426d8d62901400379e84 2023-03-08 19:00:41,827 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-2a0e9bd776cc4f468bd20a74e4f61006 2023-03-08 19:00:41,832 DEBUG [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-bd07ca0c4ad54ba0bb89f6a8507d4e9b 2023-03-08 19:00:41,832 INFO [StoreOpener-ea43d1afb394fd7a05814d2275bdeffd-1] regionserver.HStore(310): Store=ea43d1afb394fd7a05814d2275bdeffd/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:41,832 DEBUG [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-57dba886df0248f48f93263d56e12c12 2023-03-08 19:00:41,832 INFO [StoreOpener-be16768f5782c50d4c94d5fdc74d3f28-1] regionserver.HStore(310): Store=be16768f5782c50d4c94d5fdc74d3f28/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:41,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:41,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:41,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:41,833 DEBUG [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-5b72a399cd494fa4836e4a3e20c6b1ba 2023-03-08 19:00:41,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:41,834 INFO [StoreOpener-668bd868cc8589cf26791b7dda1212d8-1] regionserver.HStore(310): Store=668bd868cc8589cf26791b7dda1212d8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:41,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:41,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:41,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:41,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:41,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:41,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-03-08 19:00:41,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened ea43d1afb394fd7a05814d2275bdeffd; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73150164, jitterRate=0.09002238512039185}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:41,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for ea43d1afb394fd7a05814d2275bdeffd: 2023-03-08 19:00:41,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-03-08 19:00:41,838 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened be16768f5782c50d4c94d5fdc74d3f28; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60652999, jitterRate=-0.0961998850107193}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:41,838 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd., pid=185, masterSystemTime=1678302041811 2023-03-08 19:00:41,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for be16768f5782c50d4c94d5fdc74d3f28: 2023-03-08 19:00:41,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-03-08 19:00:41,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28., pid=184, masterSystemTime=1678302041810 2023-03-08 19:00:41,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 668bd868cc8589cf26791b7dda1212d8; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68965103, jitterRate=0.02766011655330658}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:41,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 668bd868cc8589cf26791b7dda1212d8: 2023-03-08 19:00:41,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:41,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:41,840 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:41,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''} 2023-03-08 19:00:41,840 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8., pid=187, masterSystemTime=1678302041813 2023-03-08 19:00:41,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:41,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:41,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:41,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:41,841 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:41,842 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf 2023-03-08 19:00:41,842 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:41,842 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf 2023-03-08 19:00:41,842 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302041842"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302041842"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302041842"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302041842"}]},"ts":"1678302041842"} 2023-03-08 19:00:41,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:41,842 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:41,842 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:41,842 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e3eaed3ac05b4e65948c1d4b5c75462d columnFamilyName cf 2023-03-08 19:00:41,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'} 2023-03-08 19:00:41,843 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:41,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:41,843 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302041842"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302041842"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302041842"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302041842"}]},"ts":"1678302041842"} 2023-03-08 19:00:41,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:41,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:41,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:41,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:41,843 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:41,843 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:41,843 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:41,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'} 2023-03-08 19:00:41,843 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302041843"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302041843"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302041843"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302041843"}]},"ts":"1678302041843"} 2023-03-08 19:00:41,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1678302020890 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:41,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:41,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:41,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:41,845 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=185, resume processing ppid=180 2023-03-08 19:00:41,845 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=185, ppid=180, state=SUCCESS; OpenRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,41161,1678302018011 in 183 msec 2023-03-08 19:00:41,846 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=184, resume processing ppid=177 2023-03-08 19:00:41,846 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=180, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, ASSIGN in 346 msec 2023-03-08 19:00:41,846 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=184, ppid=177, state=SUCCESS; OpenRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,46129,1678302018069 in 184 msec 2023-03-08 19:00:41,847 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=187, resume processing ppid=176 2023-03-08 19:00:41,847 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=187, ppid=176, state=SUCCESS; OpenRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939 in 182 msec 2023-03-08 19:00:41,848 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=177, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, ASSIGN in 347 msec 2023-03-08 19:00:41,848 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:41,848 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=176, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, ASSIGN in 349 msec 2023-03-08 19:00:41,848 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:41,848 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf 2023-03-08 19:00:41,848 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf 2023-03-08 19:00:41,848 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf 2023-03-08 19:00:41,849 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf 2023-03-08 19:00:41,849 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7cdf03195577655a4e5d1e7f4ff92392 columnFamilyName cf 2023-03-08 19:00:41,849 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b8add49535b4b0366d051f3a48b2a333 columnFamilyName cf 2023-03-08 19:00:41,856 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-64a5b55bb32e4bff96aba52c33665694 2023-03-08 19:00:41,862 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-657092e48c0b4ad0ab0fa0681948dafb 2023-03-08 19:00:41,865 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-172aadf8677145f0b28293dbaeda5b9f 2023-03-08 19:00:41,865 DEBUG [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-bc3874f6411e445bb582265e908e860e 2023-03-08 19:00:41,865 INFO [StoreOpener-e3eaed3ac05b4e65948c1d4b5c75462d-1] regionserver.HStore(310): Store=e3eaed3ac05b4e65948c1d4b5c75462d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:41,865 DEBUG [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-f24cdc04e3d949cf8fcfbd4389e9cdfb 2023-03-08 19:00:41,866 INFO [StoreOpener-7cdf03195577655a4e5d1e7f4ff92392-1] regionserver.HStore(310): Store=7cdf03195577655a4e5d1e7f4ff92392/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:41,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:41,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:41,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:41,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:41,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:41,869 DEBUG [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-9549786755914507847487aebb599d51 2023-03-08 19:00:41,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:41,869 INFO [StoreOpener-b8add49535b4b0366d051f3a48b2a333-1] regionserver.HStore(310): Store=b8add49535b4b0366d051f3a48b2a333/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:41,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:41,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:41,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-03-08 19:00:41,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-03-08 19:00:41,871 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened e3eaed3ac05b4e65948c1d4b5c75462d; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60049866, jitterRate=-0.10518726706504822}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:41,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for e3eaed3ac05b4e65948c1d4b5c75462d: 2023-03-08 19:00:41,871 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 7cdf03195577655a4e5d1e7f4ff92392; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74911998, jitterRate=0.11627575755119324}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:41,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 7cdf03195577655a4e5d1e7f4ff92392: 2023-03-08 19:00:41,871 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d., pid=183, masterSystemTime=1678302041811 2023-03-08 19:00:41,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392., pid=186, masterSystemTime=1678302041813 2023-03-08 19:00:41,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:41,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:41,873 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:41,873 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:41,873 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302041873"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302041873"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302041873"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302041873"}]},"ts":"1678302041873"} 2023-03-08 19:00:41,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:41,873 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:41,874 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:41,874 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302041874"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302041874"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302041874"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302041874"}]},"ts":"1678302041874"} 2023-03-08 19:00:41,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-03-08 19:00:41,875 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened b8add49535b4b0366d051f3a48b2a333; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71925670, jitterRate=0.07177600264549255}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:41,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for b8add49535b4b0366d051f3a48b2a333: 2023-03-08 19:00:41,875 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333., pid=182, masterSystemTime=1678302041810 2023-03-08 19:00:41,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:41,876 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:41,877 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:41,877 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302041877"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302041877"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302041877"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302041877"}]},"ts":"1678302041877"} 2023-03-08 19:00:41,879 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=183, resume processing ppid=181 2023-03-08 19:00:41,879 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=183, ppid=181, state=SUCCESS; OpenRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 219 msec 2023-03-08 19:00:41,879 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=186, resume processing ppid=178 2023-03-08 19:00:41,879 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=186, ppid=178, state=SUCCESS; OpenRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,33875,1678302017939 in 216 msec 2023-03-08 19:00:41,880 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=181, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, ASSIGN in 380 msec 2023-03-08 19:00:41,880 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=178, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, ASSIGN in 380 msec 2023-03-08 19:00:41,881 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=182, resume processing ppid=179 2023-03-08 19:00:41,881 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=182, ppid=179, state=SUCCESS; OpenRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,46129,1678302018069 in 222 msec 2023-03-08 19:00:41,882 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=179, resume processing ppid=175 2023-03-08 19:00:41,882 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=179, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, ASSIGN in 382 msec 2023-03-08 19:00:41,883 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302041883"}]},"ts":"1678302041883"} 2023-03-08 19:00:41,884 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=ENABLED in hbase:meta 2023-03-08 19:00:41,888 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=175, state=SUCCESS; CloneSnapshotProcedure (table=testRestoreSnapshot-1678302020890 snapshot=name: "snaptb1-1678302020890" table: "testRestoreSnapshot-1678302020890" creation_time: 1678302030666 type: DISABLED version: 2 ttl: 0 ) in 493 msec 2023-03-08 19:00:42,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-03-08 19:00:42,152 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1678302020890, procId: 175 completed 2023-03-08 19:00:42,153 DEBUG [Listener at localhost/45021] zookeeper.ReadOnlyZKClient(139): Connect 0x594888d9 to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:42,161 DEBUG [Listener at localhost/45021] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7254d96e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:42,164 DEBUG [hconnection-0x1913e027-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:42,165 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:57996, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:42,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,331 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-03-08 19:00:42,333 DEBUG [hconnection-0x1913e027-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:42,334 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47302, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:42,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,518 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-03-08 19:00:42,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,707 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-03-08 19:00:42,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,861 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-03-08 19:00:42,863 DEBUG [hconnection-0x1913e027-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:42,864 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:39400, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:42,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:42,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,058 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-03-08 19:00:43,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:43,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7201) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7367) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7137) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:44,732 INFO [Listener at localhost/45021] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1678302020890 2023-03-08 19:00:44,733 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$11(2418): Client=jenkins//172.31.2.20 disable testRestoreSnapshot-1678302020890 2023-03-08 19:00:44,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=188, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:44,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-03-08 19:00:44,736 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302044736"}]},"ts":"1678302044736"} 2023-03-08 19:00:44,737 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=DISABLING in hbase:meta 2023-03-08 19:00:44,739 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1678302020890 to state=DISABLING 2023-03-08 19:00:44,740 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN}, {pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN}, {pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN}, {pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN}, {pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN}, {pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN}] 2023-03-08 19:00:44,741 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN 2023-03-08 19:00:44,741 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN 2023-03-08 19:00:44,741 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN 2023-03-08 19:00:44,742 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN 2023-03-08 19:00:44,742 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN 2023-03-08 19:00:44,742 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:44,742 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:44,742 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302044742"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302044742"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302044742"}]},"ts":"1678302044742"} 2023-03-08 19:00:44,742 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302044742"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302044742"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302044742"}]},"ts":"1678302044742"} 2023-03-08 19:00:44,742 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:44,742 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:44,742 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302044742"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302044742"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302044742"}]},"ts":"1678302044742"} 2023-03-08 19:00:44,742 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:44,743 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302044742"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302044742"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302044742"}]},"ts":"1678302044742"} 2023-03-08 19:00:44,743 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302044742"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302044742"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302044742"}]},"ts":"1678302044742"} 2023-03-08 19:00:44,743 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=195, ppid=194, state=RUNNABLE; CloseRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:44,744 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=196, ppid=190, state=RUNNABLE; CloseRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:44,744 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=197, ppid=192, state=RUNNABLE; CloseRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:44,747 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=198, ppid=189, state=RUNNABLE; CloseRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:44,748 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=199, ppid=193, state=RUNNABLE; CloseRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:44,748 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN 2023-03-08 19:00:44,749 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:44,749 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302044749"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302044749"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302044749"}]},"ts":"1678302044749"} 2023-03-08 19:00:44,751 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=200, ppid=191, state=RUNNABLE; CloseRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:44,895 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:44,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing ea43d1afb394fd7a05814d2275bdeffd, disabling compactions & flushes 2023-03-08 19:00:44,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:44,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:44,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. after waiting 0 ms 2023-03-08 19:00:44,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:44,900 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:44,900 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:44,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 668bd868cc8589cf26791b7dda1212d8, disabling compactions & flushes 2023-03-08 19:00:44,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing be16768f5782c50d4c94d5fdc74d3f28, disabling compactions & flushes 2023-03-08 19:00:44,902 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:44,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:44,902 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:44,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. after waiting 0 ms 2023-03-08 19:00:44,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:44,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:44,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. after waiting 0 ms 2023-03-08 19:00:44,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:44,908 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-03-08 19:00:44,909 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd. 2023-03-08 19:00:44,909 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for ea43d1afb394fd7a05814d2275bdeffd: 2023-03-08 19:00:44,909 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-03-08 19:00:44,909 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-03-08 19:00:44,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8. 2023-03-08 19:00:44,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 668bd868cc8589cf26791b7dda1212d8: 2023-03-08 19:00:44,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28. 2023-03-08 19:00:44,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for be16768f5782c50d4c94d5fdc74d3f28: 2023-03-08 19:00:44,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:44,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:44,911 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing e3eaed3ac05b4e65948c1d4b5c75462d, disabling compactions & flushes 2023-03-08 19:00:44,911 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:44,911 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:44,911 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. after waiting 0 ms 2023-03-08 19:00:44,911 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:44,911 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=ea43d1afb394fd7a05814d2275bdeffd, regionState=CLOSED 2023-03-08 19:00:44,911 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302044911"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302044911"}]},"ts":"1678302044911"} 2023-03-08 19:00:44,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:44,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:44,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 7cdf03195577655a4e5d1e7f4ff92392, disabling compactions & flushes 2023-03-08 19:00:44,918 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:44,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:44,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. after waiting 0 ms 2023-03-08 19:00:44,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:44,919 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=668bd868cc8589cf26791b7dda1212d8, regionState=CLOSED 2023-03-08 19:00:44,919 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302044919"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302044919"}]},"ts":"1678302044919"} 2023-03-08 19:00:44,919 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:44,919 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:44,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing b8add49535b4b0366d051f3a48b2a333, disabling compactions & flushes 2023-03-08 19:00:44,920 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:44,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:44,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. after waiting 0 ms 2023-03-08 19:00:44,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:44,920 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=be16768f5782c50d4c94d5fdc74d3f28, regionState=CLOSED 2023-03-08 19:00:44,920 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302044920"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302044920"}]},"ts":"1678302044920"} 2023-03-08 19:00:44,923 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=199, resume processing ppid=193 2023-03-08 19:00:44,923 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=199, ppid=193, state=SUCCESS; CloseRegionProcedure ea43d1afb394fd7a05814d2275bdeffd, server=jenkins-hbase6.apache.org,41161,1678302018011 in 171 msec 2023-03-08 19:00:44,923 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-03-08 19:00:44,923 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=198, resume processing ppid=189 2023-03-08 19:00:44,924 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=198, ppid=189, state=SUCCESS; CloseRegionProcedure 668bd868cc8589cf26791b7dda1212d8, server=jenkins-hbase6.apache.org,33875,1678302017939 in 176 msec 2023-03-08 19:00:44,924 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d. 2023-03-08 19:00:44,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for e3eaed3ac05b4e65948c1d4b5c75462d: 2023-03-08 19:00:44,925 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-03-08 19:00:44,925 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333. 2023-03-08 19:00:44,925 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for b8add49535b4b0366d051f3a48b2a333: 2023-03-08 19:00:44,928 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=196, resume processing ppid=190 2023-03-08 19:00:44,929 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=193, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=ea43d1afb394fd7a05814d2275bdeffd, UNASSIGN in 183 msec 2023-03-08 19:00:44,929 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=196, ppid=190, state=SUCCESS; CloseRegionProcedure be16768f5782c50d4c94d5fdc74d3f28, server=jenkins-hbase6.apache.org,46129,1678302018069 in 178 msec 2023-03-08 19:00:44,929 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-03-08 19:00:44,929 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:44,930 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=189, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=668bd868cc8589cf26791b7dda1212d8, UNASSIGN in 184 msec 2023-03-08 19:00:44,930 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=e3eaed3ac05b4e65948c1d4b5c75462d, regionState=CLOSED 2023-03-08 19:00:44,930 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1678302044930"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302044930"}]},"ts":"1678302044930"} 2023-03-08 19:00:44,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:44,931 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=190, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=be16768f5782c50d4c94d5fdc74d3f28, UNASSIGN in 189 msec 2023-03-08 19:00:44,931 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=b8add49535b4b0366d051f3a48b2a333, regionState=CLOSED 2023-03-08 19:00:44,931 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302044931"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302044931"}]},"ts":"1678302044931"} 2023-03-08 19:00:44,933 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=195, resume processing ppid=194 2023-03-08 19:00:44,934 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=195, ppid=194, state=SUCCESS; CloseRegionProcedure e3eaed3ac05b4e65948c1d4b5c75462d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 189 msec 2023-03-08 19:00:44,934 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=197, resume processing ppid=192 2023-03-08 19:00:44,934 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=197, ppid=192, state=SUCCESS; CloseRegionProcedure b8add49535b4b0366d051f3a48b2a333, server=jenkins-hbase6.apache.org,46129,1678302018069 in 189 msec 2023-03-08 19:00:44,934 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392. 2023-03-08 19:00:44,934 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 7cdf03195577655a4e5d1e7f4ff92392: 2023-03-08 19:00:44,935 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=194, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=e3eaed3ac05b4e65948c1d4b5c75462d, UNASSIGN in 193 msec 2023-03-08 19:00:44,935 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=192, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=b8add49535b4b0366d051f3a48b2a333, UNASSIGN in 194 msec 2023-03-08 19:00:44,936 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:44,936 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=7cdf03195577655a4e5d1e7f4ff92392, regionState=CLOSED 2023-03-08 19:00:44,936 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1678302044936"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302044936"}]},"ts":"1678302044936"} 2023-03-08 19:00:44,939 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=200, resume processing ppid=191 2023-03-08 19:00:44,939 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=200, ppid=191, state=SUCCESS; CloseRegionProcedure 7cdf03195577655a4e5d1e7f4ff92392, server=jenkins-hbase6.apache.org,33875,1678302017939 in 186 msec 2023-03-08 19:00:44,940 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=191, resume processing ppid=188 2023-03-08 19:00:44,940 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=191, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1678302020890, region=7cdf03195577655a4e5d1e7f4ff92392, UNASSIGN in 199 msec 2023-03-08 19:00:44,941 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302044941"}]},"ts":"1678302044941"} 2023-03-08 19:00:44,942 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1678302020890, state=DISABLED in hbase:meta 2023-03-08 19:00:44,943 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1678302020890 to state=DISABLED 2023-03-08 19:00:44,945 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=188, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1678302020890 in 211 msec 2023-03-08 19:00:44,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-03-08 19:00:44,987 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1678302020890, procId: 188 completed 2023-03-08 19:00:44,988 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$5(2228): Client=jenkins//172.31.2.20 delete testRestoreSnapshot-1678302020890 2023-03-08 19:00:44,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:44,991 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:44,991 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=201, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:44,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-03-08 19:00:44,998 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:44,998 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:44,998 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:44,998 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:44,998 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:44,998 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:45,000 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/recovered.edits] 2023-03-08 19:00:45,000 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/recovered.edits] 2023-03-08 19:00:45,000 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/recovered.edits] 2023-03-08 19:00:45,000 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/recovered.edits] 2023-03-08 19:00:45,000 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/recovered.edits] 2023-03-08 19:00:45,001 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/recovered.edits] 2023-03-08 19:00:45,004 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-4943e5745d97426d8d62901400379e84 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-4943e5745d97426d8d62901400379e84. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041133 currentMtime: 1678302041481 2023-03-08 19:00:45,004 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-2a0e9bd776cc4f468bd20a74e4f61006 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-2a0e9bd776cc4f468bd20a74e4f61006. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041133 currentMtime: 1678302041473 2023-03-08 19:00:45,005 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-7e72d8ad17d24d9f8bfdc8e8a32a99ed already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-7e72d8ad17d24d9f8bfdc8e8a32a99ed. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041133 currentMtime: 1678302041475 2023-03-08 19:00:45,005 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-657092e48c0b4ad0ab0fa0681948dafb already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-657092e48c0b4ad0ab0fa0681948dafb. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041133 currentMtime: 1678302041482 2023-03-08 19:00:45,005 ERROR [HFileArchiver-8] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-64a5b55bb32e4bff96aba52c33665694 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-64a5b55bb32e4bff96aba52c33665694. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041133 currentMtime: 1678302041474 2023-03-08 19:00:45,005 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-4943e5745d97426d8d62901400379e84 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-4943e5745d97426d8d62901400379e84.1678302045000. 2023-03-08 19:00:45,005 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-172aadf8677145f0b28293dbaeda5b9f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-172aadf8677145f0b28293dbaeda5b9f. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041133 currentMtime: 1678302041472 2023-03-08 19:00:45,005 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-2a0e9bd776cc4f468bd20a74e4f61006 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-2a0e9bd776cc4f468bd20a74e4f61006.1678302045000. 2023-03-08 19:00:45,005 INFO [HFileArchiver-8] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-64a5b55bb32e4bff96aba52c33665694 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-64a5b55bb32e4bff96aba52c33665694.1678302045001. 2023-03-08 19:00:45,006 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-657092e48c0b4ad0ab0fa0681948dafb to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-657092e48c0b4ad0ab0fa0681948dafb.1678302045000. 2023-03-08 19:00:45,006 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-7e72d8ad17d24d9f8bfdc8e8a32a99ed to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-7e72d8ad17d24d9f8bfdc8e8a32a99ed.1678302045000. 2023-03-08 19:00:45,006 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-172aadf8677145f0b28293dbaeda5b9f to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-172aadf8677145f0b28293dbaeda5b9f.1678302045000. 2023-03-08 19:00:45,006 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-4943e5745d97426d8d62901400379e84 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-4943e5745d97426d8d62901400379e84 2023-03-08 19:00:45,008 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-64a5b55bb32e4bff96aba52c33665694 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-64a5b55bb32e4bff96aba52c33665694 2023-03-08 19:00:45,008 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-2a0e9bd776cc4f468bd20a74e4f61006 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-2a0e9bd776cc4f468bd20a74e4f61006 2023-03-08 19:00:45,008 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-7e72d8ad17d24d9f8bfdc8e8a32a99ed to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-7e72d8ad17d24d9f8bfdc8e8a32a99ed 2023-03-08 19:00:45,008 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-657092e48c0b4ad0ab0fa0681948dafb to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-657092e48c0b4ad0ab0fa0681948dafb 2023-03-08 19:00:45,008 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-172aadf8677145f0b28293dbaeda5b9f to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-172aadf8677145f0b28293dbaeda5b9f 2023-03-08 19:00:45,009 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-57dba886df0248f48f93263d56e12c12 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-57dba886df0248f48f93263d56e12c12. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041161 currentMtime: 1678302041484 2023-03-08 19:00:45,009 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-57dba886df0248f48f93263d56e12c12 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-57dba886df0248f48f93263d56e12c12.1678302045000. 2023-03-08 19:00:45,010 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-9549786755914507847487aebb599d51 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-9549786755914507847487aebb599d51. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041161 currentMtime: 1678302041474 2023-03-08 19:00:45,010 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-5b72a399cd494fa4836e4a3e20c6b1ba already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-5b72a399cd494fa4836e4a3e20c6b1ba. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041160 currentMtime: 1678302041480 2023-03-08 19:00:45,011 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-bd07ca0c4ad54ba0bb89f6a8507d4e9b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-bd07ca0c4ad54ba0bb89f6a8507d4e9b. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041160 currentMtime: 1678302041480 2023-03-08 19:00:45,011 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-9549786755914507847487aebb599d51 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-9549786755914507847487aebb599d51.1678302045000. 2023-03-08 19:00:45,011 ERROR [HFileArchiver-8] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-bc3874f6411e445bb582265e908e860e already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-bc3874f6411e445bb582265e908e860e. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041159 currentMtime: 1678302041480 2023-03-08 19:00:45,011 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-57dba886df0248f48f93263d56e12c12 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/cf/testRestoreSnapshot-1678302020890=be16768f5782c50d4c94d5fdc74d3f28-57dba886df0248f48f93263d56e12c12 2023-03-08 19:00:45,011 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-f24cdc04e3d949cf8fcfbd4389e9cdfb already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-f24cdc04e3d949cf8fcfbd4389e9cdfb. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041159 currentMtime: 1678302041484 2023-03-08 19:00:45,011 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-5b72a399cd494fa4836e4a3e20c6b1ba to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-5b72a399cd494fa4836e4a3e20c6b1ba.1678302045000. 2023-03-08 19:00:45,012 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-bd07ca0c4ad54ba0bb89f6a8507d4e9b to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-bd07ca0c4ad54ba0bb89f6a8507d4e9b.1678302045000. 2023-03-08 19:00:45,013 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-9549786755914507847487aebb599d51 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/cf/testRestoreSnapshot-1678302020890=b8add49535b4b0366d051f3a48b2a333-9549786755914507847487aebb599d51 2023-03-08 19:00:45,013 INFO [HFileArchiver-8] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-bc3874f6411e445bb582265e908e860e to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-bc3874f6411e445bb582265e908e860e.1678302045001. 2023-03-08 19:00:45,013 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-f24cdc04e3d949cf8fcfbd4389e9cdfb to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-f24cdc04e3d949cf8fcfbd4389e9cdfb.1678302045000. 2023-03-08 19:00:45,014 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-5b72a399cd494fa4836e4a3e20c6b1ba to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/cf/testRestoreSnapshot-1678302020890=668bd868cc8589cf26791b7dda1212d8-5b72a399cd494fa4836e4a3e20c6b1ba 2023-03-08 19:00:45,014 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-bd07ca0c4ad54ba0bb89f6a8507d4e9b to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/cf/testRestoreSnapshot-1678302020890=ea43d1afb394fd7a05814d2275bdeffd-bd07ca0c4ad54ba0bb89f6a8507d4e9b 2023-03-08 19:00:45,014 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-bc3874f6411e445bb582265e908e860e to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/cf/testRestoreSnapshot-1678302020890=e3eaed3ac05b4e65948c1d4b5c75462d-bc3874f6411e445bb582265e908e860e 2023-03-08 19:00:45,015 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-f24cdc04e3d949cf8fcfbd4389e9cdfb to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/cf/testRestoreSnapshot-1678302020890=7cdf03195577655a4e5d1e7f4ff92392-f24cdc04e3d949cf8fcfbd4389e9cdfb 2023-03-08 19:00:45,016 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/recovered.edits/18.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28/recovered.edits/18.seqid 2023-03-08 19:00:45,017 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/be16768f5782c50d4c94d5fdc74d3f28 2023-03-08 19:00:45,018 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/recovered.edits/18.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333/recovered.edits/18.seqid 2023-03-08 19:00:45,018 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/recovered.edits/18.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8/recovered.edits/18.seqid 2023-03-08 19:00:45,018 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/recovered.edits/18.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392/recovered.edits/18.seqid 2023-03-08 19:00:45,018 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/recovered.edits/18.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d/recovered.edits/18.seqid 2023-03-08 19:00:45,019 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/b8add49535b4b0366d051f3a48b2a333 2023-03-08 19:00:45,019 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/668bd868cc8589cf26791b7dda1212d8 2023-03-08 19:00:45,019 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/recovered.edits/18.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd/recovered.edits/18.seqid 2023-03-08 19:00:45,019 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/e3eaed3ac05b4e65948c1d4b5c75462d 2023-03-08 19:00:45,020 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/ea43d1afb394fd7a05814d2275bdeffd 2023-03-08 19:00:45,020 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testRestoreSnapshot-1678302020890/7cdf03195577655a4e5d1e7f4ff92392 2023-03-08 19:00:45,020 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1678302020890 regions 2023-03-08 19:00:45,020 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9 2023-03-08 19:00:45,021 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf] 2023-03-08 19:00:45,023 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041170 currentMtime: 1678302041416 2023-03-08 19:00:45,023 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864.1678302045021. 2023-03-08 19:00:45,024 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-a87ff679a2f3e71d9181a67b7542122c20230308780a872f376b4b86ab37768bf6922864 2023-03-08 19:00:45,025 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041171 currentMtime: 1678302041418 2023-03-08 19:00:45,025 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6.1678302045021. 2023-03-08 19:00:45,026 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-a87ff679a2f3e71d9181a67b7542122c20230308b68cc3768f364ae896849a0259b30bd6 2023-03-08 19:00:45,027 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041171 currentMtime: 1678302041420 2023-03-08 19:00:45,027 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621.1678302045021. 2023-03-08 19:00:45,028 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c4ca4238a0b923820dcc509a6f75849b20230308441393a8b34943f79b345d5b1b4ed621 2023-03-08 19:00:45,029 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041172 currentMtime: 1678302041422 2023-03-08 19:00:45,029 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4.1678302045021. 2023-03-08 19:00:45,030 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c4ca4238a0b923820dcc509a6f75849b20230308ac8f51b2237b40eb913445bab23fd7a4 2023-03-08 19:00:45,031 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041173 currentMtime: 1678302041424 2023-03-08 19:00:45,031 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954.1678302045021. 2023-03-08 19:00:45,032 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c81e728d9d4c2f636f067f89cc14862c20230308607146d900314f0c848d9f41aa8ad954 2023-03-08 19:00:45,033 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041174 currentMtime: 1678302041425 2023-03-08 19:00:45,033 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc.1678302045021. 2023-03-08 19:00:45,034 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-c81e728d9d4c2f636f067f89cc14862c20230308d12551c3c0e14678a07028e4e728aafc 2023-03-08 19:00:45,035 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041175 currentMtime: 1678302041427 2023-03-08 19:00:45,035 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d.1678302045021. 2023-03-08 19:00:45,036 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-d41d8cd98f00b204e9800998ecf8427e202303087629e8aefe8b47668a107698c4f2e71d 2023-03-08 19:00:45,036 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041176 currentMtime: 1678302041428 2023-03-08 19:00:45,037 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279.1678302045021. 2023-03-08 19:00:45,037 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-d41d8cd98f00b204e9800998ecf8427e20230308ec1b26c69af04935b76efbe8450fc279 2023-03-08 19:00:45,038 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041177 currentMtime: 1678302041430 2023-03-08 19:00:45,039 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416.1678302045021. 2023-03-08 19:00:45,039 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-e4da3b7fbbce2345d7772b0674a318d52023030869a46b51233d40d0828841564d7cb416 2023-03-08 19:00:45,040 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041178 currentMtime: 1678302041431 2023-03-08 19:00:45,040 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6.1678302045021. 2023-03-08 19:00:45,041 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-e4da3b7fbbce2345d7772b0674a318d520230308aaddf52827d149e89d1efcfcc25a09a6 2023-03-08 19:00:45,042 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041179 currentMtime: 1678302041433 2023-03-08 19:00:45,042 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90.1678302045021. 2023-03-08 19:00:45,043 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-eccbc87e4b5ce2fe28308fd9f2a7baf320230308850b055ae0f34e94bcec8fb6f1e9fb90 2023-03-08 19:00:45,044 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf. archiveLen: 0 currentLen: 0 archiveMtime: 1678302041180 currentMtime: 1678302041435 2023-03-08 19:00:45,044 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf.1678302045021. 2023-03-08 19:00:45,045 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9/cf/testRestoreSnapshot-1678302020890=f53d35b3eecf0fe701345a9b83dc29c9-eccbc87e4b5ce2fe28308fd9f2a7baf32023030898fdb4b8adc04cca80dbd5d35db1fbbf 2023-03-08 19:00:45,045 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testRestoreSnapshot-1678302020890/f53d35b3eecf0fe701345a9b83dc29c9 2023-03-08 19:00:45,047 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=201, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:45,050 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1678302020890 from hbase:meta 2023-03-08 19:00:45,051 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1678302020890' descriptor. 2023-03-08 19:00:45,052 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=201, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:45,052 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1678302020890' from region states. 2023-03-08 19:00:45,052 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302045052"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:45,053 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302045052"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:45,053 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302045052"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:45,053 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302045052"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:45,053 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302045052"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:45,053 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302045052"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:45,054 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-03-08 19:00:45,054 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 668bd868cc8589cf26791b7dda1212d8, NAME => 'testRestoreSnapshot-1678302020890,,1678302020918.668bd868cc8589cf26791b7dda1212d8.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => be16768f5782c50d4c94d5fdc74d3f28, NAME => 'testRestoreSnapshot-1678302020890,1,1678302020918.be16768f5782c50d4c94d5fdc74d3f28.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 7cdf03195577655a4e5d1e7f4ff92392, NAME => 'testRestoreSnapshot-1678302020890,2,1678302020918.7cdf03195577655a4e5d1e7f4ff92392.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => b8add49535b4b0366d051f3a48b2a333, NAME => 'testRestoreSnapshot-1678302020890,3,1678302020918.b8add49535b4b0366d051f3a48b2a333.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => ea43d1afb394fd7a05814d2275bdeffd, NAME => 'testRestoreSnapshot-1678302020890,4,1678302020918.ea43d1afb394fd7a05814d2275bdeffd.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => e3eaed3ac05b4e65948c1d4b5c75462d, NAME => 'testRestoreSnapshot-1678302020890,5,1678302020918.e3eaed3ac05b4e65948c1d4b5c75462d.', STARTKEY => '5', ENDKEY => ''}] 2023-03-08 19:00:45,054 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1678302020890' as deleted. 2023-03-08 19:00:45,054 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1678302020890","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1678302045054"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:45,056 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1678302020890 state from META 2023-03-08 19:00:45,058 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=201, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1678302020890 2023-03-08 19:00:45,058 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=201, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1678302020890 in 70 msec 2023-03-08 19:00:45,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-03-08 19:00:45,243 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1678302020890, procId: 201 completed 2023-03-08 19:00:45,249 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(748): Client=jenkins//172.31.2.20 delete name: "emptySnaptb-1678302020890" 2023-03-08 19:00:45,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1678302020890 2023-03-08 19:00:45,251 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(748): Client=jenkins//172.31.2.20 delete name: "snaptb0-1678302020890" 2023-03-08 19:00:45,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1678302020890 2023-03-08 19:00:45,253 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(748): Client=jenkins//172.31.2.20 delete name: "snaptb1-1678302020890" 2023-03-08 19:00:45,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(333): Deleting snapshot: snaptb1-1678302020890 2023-03-08 19:00:45,280 INFO [Listener at localhost/45021] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=428 (was 362) Potentially hanging thread: hconnection-0x6f162865-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-959595253_15 at /127.0.0.1:41150 [Waiting for operation #1020] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-717524908_15 at /127.0.0.1:60698 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/cluster_ea2ca4d9-6781-beee-9c1a-51bfaa940573/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-717524908_15 at /127.0.0.1:55836 [Waiting for operation #756] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-959595253_15 at /127.0.0.1:40946 [Waiting for operation #277] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1386908842_15 at /127.0.0.1:60712 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-metaLookup-shared--pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-959595253_15 at /127.0.0.1:41248 [Waiting for operation #473] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:59169@0x594888d9-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-717524908_15 at /127.0.0.1:60574 [Waiting for operation #478] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1386908842_15 at /127.0.0.1:41212 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:59169@0x594888d9-SendThread(127.0.0.1:59169) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: hconnection-0x6f162865-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-shared-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6f162865-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/cluster_ea2ca4d9-6781-beee-9c1a-51bfaa940573/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/cluster_ea2ca4d9-6781-beee-9c1a-51bfaa940573/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3fa82af-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-959595253_15 at /127.0.0.1:41238 [Waiting for operation #500] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-959595253_15 at /127.0.0.1:41134 [Waiting for operation #1013] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/cluster_ea2ca4d9-6781-beee-9c1a-51bfaa940573/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/cluster_ea2ca4d9-6781-beee-9c1a-51bfaa940573/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-metaLookup-shared--pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-751704260_15 at /127.0.0.1:60702 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-751704260_15 at /127.0.0.1:60408 [Waiting for operation #247] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-shared-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/cluster_ea2ca4d9-6781-beee-9c1a-51bfaa940573/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-metaLookup-shared--pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6f162865-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1386908842_15 at /127.0.0.1:60682 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-metaLookup-shared--pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-shared-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-717524908_15 at /127.0.0.1:60666 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-751704260_15 at /127.0.0.1:60672 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-959595253_15 at /127.0.0.1:46766 [Waiting for operation #1045] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:59169@0x594888d9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$45/247024778.run(Unknown Source) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=814 (was 695) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=314 (was 391), ProcessCount=170 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=7488 (was 7283) - AvailableMemoryMB LEAK? - 2023-03-08 19:00:45,295 INFO [Listener at localhost/45021] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=428, OpenFileDescriptor=814, MaxFileDescriptor=60000, SystemLoadAverage=314, ProcessCount=170, AvailableMemoryMB=7486 2023-03-08 19:00:45,297 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$4(2112): Client=jenkins//172.31.2.20 create 'testCorruptedSnapshot-1678302045295', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-03-08 19:00:45,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCorruptedSnapshot-1678302045295 2023-03-08 19:00:45,299 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1678302045295 execute state=CREATE_TABLE_PRE_OPERATION 2023-03-08 19:00:45,300 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(697): Client=jenkins//172.31.2.20 procedure request for creating table: namespace: "default" qualifier: "testCorruptedSnapshot-1678302045295" procId is: 202 2023-03-08 19:00:45,300 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1678302045295 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-08 19:00:45,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-03-08 19:00:45,304 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:45,304 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:45,304 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:45,304 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:45,304 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:45,304 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:45,304 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d empty. 2023-03-08 19:00:45,304 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c empty. 2023-03-08 19:00:45,304 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e empty. 2023-03-08 19:00:45,304 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c empty. 2023-03-08 19:00:45,304 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d empty. 2023-03-08 19:00:45,306 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:45,306 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62 empty. 2023-03-08 19:00:45,306 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:45,306 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:45,306 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:45,306 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:45,306 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:45,306 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1678302045295 regions 2023-03-08 19:00:45,316 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/.tabledesc/.tableinfo.0000000001 2023-03-08 19:00:45,318 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(7675): creating {ENCODED => 43584d4ed824c092029e275dd3b82a6c, NAME => 'testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testCorruptedSnapshot-1678302045295', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:45,318 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(7675): creating {ENCODED => f7a0b9e431b1811a0203970d08f9a42d, NAME => 'testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testCorruptedSnapshot-1678302045295', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:45,318 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(7675): creating {ENCODED => f98146aa6eba5589f1036e30131dab62, NAME => 'testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testCorruptedSnapshot-1678302045295', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:45,332 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:45,332 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(1604): Closing 43584d4ed824c092029e275dd3b82a6c, disabling compactions & flushes 2023-03-08 19:00:45,332 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:45,332 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:45,332 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. after waiting 0 ms 2023-03-08 19:00:45,332 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:45,332 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:45,332 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(1558): Region close journal for 43584d4ed824c092029e275dd3b82a6c: 2023-03-08 19:00:45,333 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(7675): creating {ENCODED => 5b3ed2989f5355f239709c8420b1a06c, NAME => 'testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testCorruptedSnapshot-1678302045295', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:45,333 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:45,333 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(1604): Closing f7a0b9e431b1811a0203970d08f9a42d, disabling compactions & flushes 2023-03-08 19:00:45,333 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:45,333 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:45,333 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. after waiting 0 ms 2023-03-08 19:00:45,333 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:45,334 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:45,334 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(1558): Region close journal for f7a0b9e431b1811a0203970d08f9a42d: 2023-03-08 19:00:45,334 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4446f78ec8b0f1645c8eb2d6d276d62d, NAME => 'testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testCorruptedSnapshot-1678302045295', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:45,334 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:45,334 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(1604): Closing f98146aa6eba5589f1036e30131dab62, disabling compactions & flushes 2023-03-08 19:00:45,334 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:45,334 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:45,334 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. after waiting 0 ms 2023-03-08 19:00:45,334 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:45,334 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:45,334 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(1558): Region close journal for f98146aa6eba5589f1036e30131dab62: 2023-03-08 19:00:45,335 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(7675): creating {ENCODED => a8e2ccd674e201b22ff6152ea57faf8e, NAME => 'testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testCorruptedSnapshot-1678302045295', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp 2023-03-08 19:00:45,350 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:45,350 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:45,350 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(1604): Closing 5b3ed2989f5355f239709c8420b1a06c, disabling compactions & flushes 2023-03-08 19:00:45,350 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(1604): Closing 4446f78ec8b0f1645c8eb2d6d276d62d, disabling compactions & flushes 2023-03-08 19:00:45,350 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:45,350 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:45,350 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:45,350 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:45,350 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. after waiting 0 ms 2023-03-08 19:00:45,350 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:45,350 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. after waiting 0 ms 2023-03-08 19:00:45,350 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:45,350 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:45,350 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:45,350 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-2] regionserver.HRegion(1558): Region close journal for 5b3ed2989f5355f239709c8420b1a06c: 2023-03-08 19:00:45,350 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-0] regionserver.HRegion(1558): Region close journal for 4446f78ec8b0f1645c8eb2d6d276d62d: 2023-03-08 19:00:45,352 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:45,352 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(1604): Closing a8e2ccd674e201b22ff6152ea57faf8e, disabling compactions & flushes 2023-03-08 19:00:45,352 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:45,352 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:45,352 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. after waiting 0 ms 2023-03-08 19:00:45,352 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:45,352 INFO [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:45,352 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1678302045295-pool-1] regionserver.HRegion(1558): Region close journal for a8e2ccd674e201b22ff6152ea57faf8e: 2023-03-08 19:00:45,354 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1678302045295 execute state=CREATE_TABLE_ADD_TO_META 2023-03-08 19:00:45,355 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302045355"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302045355"}]},"ts":"1678302045355"} 2023-03-08 19:00:45,355 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302045355"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302045355"}]},"ts":"1678302045355"} 2023-03-08 19:00:45,355 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302045355"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302045355"}]},"ts":"1678302045355"} 2023-03-08 19:00:45,355 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302045355"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302045355"}]},"ts":"1678302045355"} 2023-03-08 19:00:45,355 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302045355"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302045355"}]},"ts":"1678302045355"} 2023-03-08 19:00:45,355 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302045355"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302045355"}]},"ts":"1678302045355"} 2023-03-08 19:00:45,359 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-03-08 19:00:45,360 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1678302045295 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-08 19:00:45,360 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302045360"}]},"ts":"1678302045360"} 2023-03-08 19:00:45,361 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1678302045295, state=ENABLING in hbase:meta 2023-03-08 19:00:45,366 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase6.apache.org=0} racks are {/default-rack=0} 2023-03-08 19:00:45,368 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-08 19:00:45,368 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-03-08 19:00:45,368 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-03-08 19:00:45,368 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-03-08 19:00:45,368 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, ASSIGN}, {pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, ASSIGN}, {pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, ASSIGN}, {pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, ASSIGN}, {pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, ASSIGN}, {pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, ASSIGN}] 2023-03-08 19:00:45,370 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, ASSIGN 2023-03-08 19:00:45,370 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, ASSIGN 2023-03-08 19:00:45,370 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, ASSIGN 2023-03-08 19:00:45,370 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, ASSIGN 2023-03-08 19:00:45,371 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, ASSIGN 2023-03-08 19:00:45,371 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,46129,1678302018069; forceNewPlan=false, retain=false 2023-03-08 19:00:45,371 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,46129,1678302018069; forceNewPlan=false, retain=false 2023-03-08 19:00:45,371 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,33875,1678302017939; forceNewPlan=false, retain=false 2023-03-08 19:00:45,371 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,41161,1678302018011; forceNewPlan=false, retain=false 2023-03-08 19:00:45,371 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,41161,1678302018011; forceNewPlan=false, retain=false 2023-03-08 19:00:45,372 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, ASSIGN 2023-03-08 19:00:45,372 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,33875,1678302017939; forceNewPlan=false, retain=false 2023-03-08 19:00:45,521 INFO [jenkins-hbase6:43879] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-03-08 19:00:45,524 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=f98146aa6eba5589f1036e30131dab62, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:45,524 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=4446f78ec8b0f1645c8eb2d6d276d62d, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:45,524 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302045524"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302045524"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302045524"}]},"ts":"1678302045524"} 2023-03-08 19:00:45,524 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=a8e2ccd674e201b22ff6152ea57faf8e, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:45,524 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=43584d4ed824c092029e275dd3b82a6c, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:45,524 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=f7a0b9e431b1811a0203970d08f9a42d, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:45,524 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302045524"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302045524"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302045524"}]},"ts":"1678302045524"} 2023-03-08 19:00:45,524 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302045524"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302045524"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302045524"}]},"ts":"1678302045524"} 2023-03-08 19:00:45,524 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302045524"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302045524"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302045524"}]},"ts":"1678302045524"} 2023-03-08 19:00:45,524 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302045524"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302045524"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302045524"}]},"ts":"1678302045524"} 2023-03-08 19:00:45,526 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=209, ppid=204, state=RUNNABLE; OpenRegionProcedure f98146aa6eba5589f1036e30131dab62, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:45,526 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=210, ppid=205, state=RUNNABLE; OpenRegionProcedure 43584d4ed824c092029e275dd3b82a6c, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:45,527 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=211, ppid=203, state=RUNNABLE; OpenRegionProcedure f7a0b9e431b1811a0203970d08f9a42d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:45,527 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=212, ppid=208, state=RUNNABLE; OpenRegionProcedure a8e2ccd674e201b22ff6152ea57faf8e, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:45,528 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=213, ppid=207, state=RUNNABLE; OpenRegionProcedure 4446f78ec8b0f1645c8eb2d6d276d62d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:45,528 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=5b3ed2989f5355f239709c8420b1a06c, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:45,528 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302045528"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302045528"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302045528"}]},"ts":"1678302045528"} 2023-03-08 19:00:45,529 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=214, ppid=206, state=RUNNABLE; OpenRegionProcedure 5b3ed2989f5355f239709c8420b1a06c, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:45,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-03-08 19:00:45,679 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:45,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f98146aa6eba5589f1036e30131dab62, NAME => 'testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.', STARTKEY => '1', ENDKEY => '2'} 2023-03-08 19:00:45,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:45,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:45,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:45,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:45,680 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:45,680 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:45,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4446f78ec8b0f1645c8eb2d6d276d62d, NAME => 'testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.', STARTKEY => '4', ENDKEY => '5'} 2023-03-08 19:00:45,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5b3ed2989f5355f239709c8420b1a06c, NAME => 'testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.', STARTKEY => '3', ENDKEY => '4'} 2023-03-08 19:00:45,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:45,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:45,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:45,681 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:45,681 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:45,681 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:45,681 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:45,681 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:45,681 INFO [StoreOpener-f98146aa6eba5589f1036e30131dab62-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:45,682 INFO [StoreOpener-4446f78ec8b0f1645c8eb2d6d276d62d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:45,682 INFO [StoreOpener-5b3ed2989f5355f239709c8420b1a06c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:45,682 DEBUG [StoreOpener-f98146aa6eba5589f1036e30131dab62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf 2023-03-08 19:00:45,682 DEBUG [StoreOpener-f98146aa6eba5589f1036e30131dab62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf 2023-03-08 19:00:45,683 INFO [StoreOpener-f98146aa6eba5589f1036e30131dab62-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f98146aa6eba5589f1036e30131dab62 columnFamilyName cf 2023-03-08 19:00:45,683 DEBUG [StoreOpener-4446f78ec8b0f1645c8eb2d6d276d62d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf 2023-03-08 19:00:45,683 DEBUG [StoreOpener-5b3ed2989f5355f239709c8420b1a06c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf 2023-03-08 19:00:45,683 DEBUG [StoreOpener-5b3ed2989f5355f239709c8420b1a06c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf 2023-03-08 19:00:45,683 DEBUG [StoreOpener-4446f78ec8b0f1645c8eb2d6d276d62d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf 2023-03-08 19:00:45,683 INFO [StoreOpener-5b3ed2989f5355f239709c8420b1a06c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5b3ed2989f5355f239709c8420b1a06c columnFamilyName cf 2023-03-08 19:00:45,683 INFO [StoreOpener-4446f78ec8b0f1645c8eb2d6d276d62d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4446f78ec8b0f1645c8eb2d6d276d62d columnFamilyName cf 2023-03-08 19:00:45,684 INFO [StoreOpener-f98146aa6eba5589f1036e30131dab62-1] regionserver.HStore(310): Store=f98146aa6eba5589f1036e30131dab62/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:45,684 INFO [StoreOpener-5b3ed2989f5355f239709c8420b1a06c-1] regionserver.HStore(310): Store=5b3ed2989f5355f239709c8420b1a06c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:45,684 INFO [StoreOpener-4446f78ec8b0f1645c8eb2d6d276d62d-1] regionserver.HStore(310): Store=4446f78ec8b0f1645c8eb2d6d276d62d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:45,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:45,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:45,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:45,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:45,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:45,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:45,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:45,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:45,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:45,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-08 19:00:45,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-08 19:00:45,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-08 19:00:45,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 4446f78ec8b0f1645c8eb2d6d276d62d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71132384, jitterRate=0.05995512008666992}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:45,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 5b3ed2989f5355f239709c8420b1a06c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66920722, jitterRate=-0.0028035342693328857}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:45,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 4446f78ec8b0f1645c8eb2d6d276d62d: 2023-03-08 19:00:45,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 5b3ed2989f5355f239709c8420b1a06c: 2023-03-08 19:00:45,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened f98146aa6eba5589f1036e30131dab62; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73038351, jitterRate=0.08835624158382416}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:45,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for f98146aa6eba5589f1036e30131dab62: 2023-03-08 19:00:45,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62., pid=209, masterSystemTime=1678302045677 2023-03-08 19:00:45,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c., pid=214, masterSystemTime=1678302045678 2023-03-08 19:00:45,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d., pid=213, masterSystemTime=1678302045678 2023-03-08 19:00:45,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:45,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:45,692 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:45,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f7a0b9e431b1811a0203970d08f9a42d, NAME => 'testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.', STARTKEY => '', ENDKEY => '1'} 2023-03-08 19:00:45,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:45,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:45,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:45,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:45,692 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=4446f78ec8b0f1645c8eb2d6d276d62d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:45,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:45,692 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302045692"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302045692"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302045692"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302045692"}]},"ts":"1678302045692"} 2023-03-08 19:00:45,693 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:45,693 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:45,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 43584d4ed824c092029e275dd3b82a6c, NAME => 'testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.', STARTKEY => '2', ENDKEY => '3'} 2023-03-08 19:00:45,693 INFO [StoreOpener-f7a0b9e431b1811a0203970d08f9a42d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:45,693 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=5b3ed2989f5355f239709c8420b1a06c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:45,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:45,693 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302045693"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302045693"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302045693"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302045693"}]},"ts":"1678302045693"} 2023-03-08 19:00:45,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:45,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:45,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:45,693 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:45,693 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:45,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:45,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a8e2ccd674e201b22ff6152ea57faf8e, NAME => 'testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.', STARTKEY => '5', ENDKEY => ''} 2023-03-08 19:00:45,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:45,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:45,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:45,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:45,694 DEBUG [StoreOpener-f7a0b9e431b1811a0203970d08f9a42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf 2023-03-08 19:00:45,694 DEBUG [StoreOpener-f7a0b9e431b1811a0203970d08f9a42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf 2023-03-08 19:00:45,694 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=f98146aa6eba5589f1036e30131dab62, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:45,694 INFO [StoreOpener-f7a0b9e431b1811a0203970d08f9a42d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f7a0b9e431b1811a0203970d08f9a42d columnFamilyName cf 2023-03-08 19:00:45,694 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302045694"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302045694"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302045694"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302045694"}]},"ts":"1678302045694"} 2023-03-08 19:00:45,695 INFO [StoreOpener-a8e2ccd674e201b22ff6152ea57faf8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:45,695 INFO [StoreOpener-f7a0b9e431b1811a0203970d08f9a42d-1] regionserver.HStore(310): Store=f7a0b9e431b1811a0203970d08f9a42d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:45,696 INFO [StoreOpener-43584d4ed824c092029e275dd3b82a6c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:45,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:45,696 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=213, resume processing ppid=207 2023-03-08 19:00:45,696 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=213, ppid=207, state=SUCCESS; OpenRegionProcedure 4446f78ec8b0f1645c8eb2d6d276d62d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 166 msec 2023-03-08 19:00:45,697 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:45,697 DEBUG [StoreOpener-a8e2ccd674e201b22ff6152ea57faf8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf 2023-03-08 19:00:45,697 DEBUG [StoreOpener-a8e2ccd674e201b22ff6152ea57faf8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf 2023-03-08 19:00:45,698 INFO [StoreOpener-a8e2ccd674e201b22ff6152ea57faf8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a8e2ccd674e201b22ff6152ea57faf8e columnFamilyName cf 2023-03-08 19:00:45,698 DEBUG [StoreOpener-43584d4ed824c092029e275dd3b82a6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf 2023-03-08 19:00:45,698 DEBUG [StoreOpener-43584d4ed824c092029e275dd3b82a6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf 2023-03-08 19:00:45,698 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=214, resume processing ppid=206 2023-03-08 19:00:45,698 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=214, ppid=206, state=SUCCESS; OpenRegionProcedure 5b3ed2989f5355f239709c8420b1a06c, server=jenkins-hbase6.apache.org,33875,1678302017939 in 166 msec 2023-03-08 19:00:45,698 INFO [StoreOpener-43584d4ed824c092029e275dd3b82a6c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 43584d4ed824c092029e275dd3b82a6c columnFamilyName cf 2023-03-08 19:00:45,699 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=207, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, ASSIGN in 328 msec 2023-03-08 19:00:45,699 INFO [StoreOpener-a8e2ccd674e201b22ff6152ea57faf8e-1] regionserver.HStore(310): Store=a8e2ccd674e201b22ff6152ea57faf8e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:45,699 INFO [StoreOpener-43584d4ed824c092029e275dd3b82a6c-1] regionserver.HStore(310): Store=43584d4ed824c092029e275dd3b82a6c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:45,699 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=209, resume processing ppid=204 2023-03-08 19:00:45,700 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=206, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, ASSIGN in 330 msec 2023-03-08 19:00:45,700 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=209, ppid=204, state=SUCCESS; OpenRegionProcedure f98146aa6eba5589f1036e30131dab62, server=jenkins-hbase6.apache.org,46129,1678302018069 in 172 msec 2023-03-08 19:00:45,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:45,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:45,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:45,701 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:45,701 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:45,701 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=204, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, ASSIGN in 332 msec 2023-03-08 19:00:45,702 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-08 19:00:45,703 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened f7a0b9e431b1811a0203970d08f9a42d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62816610, jitterRate=-0.06395956873893738}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:45,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for f7a0b9e431b1811a0203970d08f9a42d: 2023-03-08 19:00:45,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:45,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:45,703 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d., pid=211, masterSystemTime=1678302045678 2023-03-08 19:00:45,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:45,705 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:45,705 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=f7a0b9e431b1811a0203970d08f9a42d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:45,705 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302045705"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302045705"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302045705"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302045705"}]},"ts":"1678302045705"} 2023-03-08 19:00:45,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-08 19:00:45,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-08 19:00:45,707 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=211, resume processing ppid=203 2023-03-08 19:00:45,708 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened a8e2ccd674e201b22ff6152ea57faf8e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64507445, jitterRate=-0.038764163851737976}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:45,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for a8e2ccd674e201b22ff6152ea57faf8e: 2023-03-08 19:00:45,708 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=211, ppid=203, state=SUCCESS; OpenRegionProcedure f7a0b9e431b1811a0203970d08f9a42d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 179 msec 2023-03-08 19:00:45,708 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 43584d4ed824c092029e275dd3b82a6c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75324457, jitterRate=0.12242187559604645}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:45,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 43584d4ed824c092029e275dd3b82a6c: 2023-03-08 19:00:45,708 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e., pid=212, masterSystemTime=1678302045677 2023-03-08 19:00:45,708 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c., pid=210, masterSystemTime=1678302045678 2023-03-08 19:00:45,709 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=203, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, ASSIGN in 340 msec 2023-03-08 19:00:45,709 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:45,709 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:45,710 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=a8e2ccd674e201b22ff6152ea57faf8e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:45,710 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302045710"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302045710"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302045710"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302045710"}]},"ts":"1678302045710"} 2023-03-08 19:00:45,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:45,710 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:45,710 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=43584d4ed824c092029e275dd3b82a6c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:45,711 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302045710"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302045710"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302045710"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302045710"}]},"ts":"1678302045710"} 2023-03-08 19:00:45,712 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=212, resume processing ppid=208 2023-03-08 19:00:45,712 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=212, ppid=208, state=SUCCESS; OpenRegionProcedure a8e2ccd674e201b22ff6152ea57faf8e, server=jenkins-hbase6.apache.org,46129,1678302018069 in 184 msec 2023-03-08 19:00:45,713 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=210, resume processing ppid=205 2023-03-08 19:00:45,713 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=210, ppid=205, state=SUCCESS; OpenRegionProcedure 43584d4ed824c092029e275dd3b82a6c, server=jenkins-hbase6.apache.org,33875,1678302017939 in 186 msec 2023-03-08 19:00:45,713 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=208, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, ASSIGN in 344 msec 2023-03-08 19:00:45,714 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=205, resume processing ppid=202 2023-03-08 19:00:45,714 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=205, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, ASSIGN in 345 msec 2023-03-08 19:00:45,715 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1678302045295 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-08 19:00:45,715 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302045715"}]},"ts":"1678302045715"} 2023-03-08 19:00:45,716 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1678302045295, state=ENABLED in hbase:meta 2023-03-08 19:00:45,721 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1678302045295 execute state=CREATE_TABLE_POST_OPERATION 2023-03-08 19:00:45,722 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=202, state=SUCCESS; CreateTableProcedure table=testCorruptedSnapshot-1678302045295 in 424 msec 2023-03-08 19:00:46,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-03-08 19:00:46,052 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCorruptedSnapshot-1678302045295, procId: 202 completed 2023-03-08 19:00:46,055 DEBUG [Listener at localhost/45021] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1678302045295 2023-03-08 19:00:46,055 DEBUG [Listener at localhost/45021] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:46,058 INFO [Listener at localhost/45021] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-03-08 19:00:46,070 INFO [Listener at localhost/45021] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1678302045295 2023-03-08 19:00:46,070 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$11(2418): Client=jenkins//172.31.2.20 disable testCorruptedSnapshot-1678302045295 2023-03-08 19:00:46,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=215, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1678302045295 2023-03-08 19:00:46,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-03-08 19:00:46,073 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302046073"}]},"ts":"1678302046073"} 2023-03-08 19:00:46,074 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1678302045295, state=DISABLING in hbase:meta 2023-03-08 19:00:46,076 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1678302045295 to state=DISABLING 2023-03-08 19:00:46,077 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, UNASSIGN}, {pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, UNASSIGN}, {pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, UNASSIGN}, {pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, UNASSIGN}, {pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, UNASSIGN}, {pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, UNASSIGN}] 2023-03-08 19:00:46,079 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, UNASSIGN 2023-03-08 19:00:46,079 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, UNASSIGN 2023-03-08 19:00:46,079 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, UNASSIGN 2023-03-08 19:00:46,079 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, UNASSIGN 2023-03-08 19:00:46,080 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, UNASSIGN 2023-03-08 19:00:46,080 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=43584d4ed824c092029e275dd3b82a6c, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:46,080 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=4446f78ec8b0f1645c8eb2d6d276d62d, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:46,080 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=5b3ed2989f5355f239709c8420b1a06c, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:46,080 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302046080"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302046080"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302046080"}]},"ts":"1678302046080"} 2023-03-08 19:00:46,080 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302046080"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302046080"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302046080"}]},"ts":"1678302046080"} 2023-03-08 19:00:46,080 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302046080"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302046080"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302046080"}]},"ts":"1678302046080"} 2023-03-08 19:00:46,081 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=f98146aa6eba5589f1036e30131dab62, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:46,081 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=a8e2ccd674e201b22ff6152ea57faf8e, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:46,081 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302046080"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302046080"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302046080"}]},"ts":"1678302046080"} 2023-03-08 19:00:46,081 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302046081"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302046081"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302046081"}]},"ts":"1678302046081"} 2023-03-08 19:00:46,081 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=222, ppid=218, state=RUNNABLE; CloseRegionProcedure 43584d4ed824c092029e275dd3b82a6c, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:46,082 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=223, ppid=219, state=RUNNABLE; CloseRegionProcedure 5b3ed2989f5355f239709c8420b1a06c, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:46,083 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=224, ppid=220, state=RUNNABLE; CloseRegionProcedure 4446f78ec8b0f1645c8eb2d6d276d62d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:46,084 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=225, ppid=217, state=RUNNABLE; CloseRegionProcedure f98146aa6eba5589f1036e30131dab62, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:46,084 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=226, ppid=221, state=RUNNABLE; CloseRegionProcedure a8e2ccd674e201b22ff6152ea57faf8e, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:46,084 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, UNASSIGN 2023-03-08 19:00:46,087 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=f7a0b9e431b1811a0203970d08f9a42d, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:46,087 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302046087"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302046087"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302046087"}]},"ts":"1678302046087"} 2023-03-08 19:00:46,088 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=227, ppid=216, state=RUNNABLE; CloseRegionProcedure f7a0b9e431b1811a0203970d08f9a42d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:46,234 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:46,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 5b3ed2989f5355f239709c8420b1a06c, disabling compactions & flushes 2023-03-08 19:00:46,234 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:46,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:46,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. after waiting 0 ms 2023-03-08 19:00:46,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:46,235 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:46,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 4446f78ec8b0f1645c8eb2d6d276d62d, disabling compactions & flushes 2023-03-08 19:00:46,235 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:46,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:46,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. after waiting 0 ms 2023-03-08 19:00:46,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:46,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-08 19:00:46,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-08 19:00:46,238 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:46,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing f98146aa6eba5589f1036e30131dab62, disabling compactions & flushes 2023-03-08 19:00:46,238 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:46,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:46,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. after waiting 0 ms 2023-03-08 19:00:46,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:46,238 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:46,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 5b3ed2989f5355f239709c8420b1a06c: 2023-03-08 19:00:46,238 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:46,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 4446f78ec8b0f1645c8eb2d6d276d62d: 2023-03-08 19:00:46,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:46,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:46,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 43584d4ed824c092029e275dd3b82a6c, disabling compactions & flushes 2023-03-08 19:00:46,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:46,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:46,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. after waiting 0 ms 2023-03-08 19:00:46,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:46,240 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=5b3ed2989f5355f239709c8420b1a06c, regionState=CLOSED 2023-03-08 19:00:46,240 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302046240"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302046240"}]},"ts":"1678302046240"} 2023-03-08 19:00:46,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:46,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:46,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing f7a0b9e431b1811a0203970d08f9a42d, disabling compactions & flushes 2023-03-08 19:00:46,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:46,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:46,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. after waiting 0 ms 2023-03-08 19:00:46,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:46,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-08 19:00:46,241 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=4446f78ec8b0f1645c8eb2d6d276d62d, regionState=CLOSED 2023-03-08 19:00:46,241 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302046241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302046241"}]},"ts":"1678302046241"} 2023-03-08 19:00:46,242 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:46,242 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for f98146aa6eba5589f1036e30131dab62: 2023-03-08 19:00:46,246 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=223, resume processing ppid=219 2023-03-08 19:00:46,247 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=223, ppid=219, state=SUCCESS; CloseRegionProcedure 5b3ed2989f5355f239709c8420b1a06c, server=jenkins-hbase6.apache.org,33875,1678302017939 in 159 msec 2023-03-08 19:00:46,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:46,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:46,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing a8e2ccd674e201b22ff6152ea57faf8e, disabling compactions & flushes 2023-03-08 19:00:46,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:46,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-08 19:00:46,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:46,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. after waiting 0 ms 2023-03-08 19:00:46,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:46,247 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=f98146aa6eba5589f1036e30131dab62, regionState=CLOSED 2023-03-08 19:00:46,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:46,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 43584d4ed824c092029e275dd3b82a6c: 2023-03-08 19:00:46,247 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302046247"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302046247"}]},"ts":"1678302046247"} 2023-03-08 19:00:46,248 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=224, resume processing ppid=220 2023-03-08 19:00:46,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-08 19:00:46,248 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=224, ppid=220, state=SUCCESS; CloseRegionProcedure 4446f78ec8b0f1645c8eb2d6d276d62d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 160 msec 2023-03-08 19:00:46,248 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=219, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, UNASSIGN in 170 msec 2023-03-08 19:00:46,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:46,249 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=43584d4ed824c092029e275dd3b82a6c, regionState=CLOSED 2023-03-08 19:00:46,250 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302046249"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302046249"}]},"ts":"1678302046249"} 2023-03-08 19:00:46,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:46,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for f7a0b9e431b1811a0203970d08f9a42d: 2023-03-08 19:00:46,250 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=220, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, UNASSIGN in 171 msec 2023-03-08 19:00:46,251 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:46,251 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=225, resume processing ppid=217 2023-03-08 19:00:46,251 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=225, ppid=217, state=SUCCESS; CloseRegionProcedure f98146aa6eba5589f1036e30131dab62, server=jenkins-hbase6.apache.org,46129,1678302018069 in 165 msec 2023-03-08 19:00:46,252 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=f7a0b9e431b1811a0203970d08f9a42d, regionState=CLOSED 2023-03-08 19:00:46,252 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302046252"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302046252"}]},"ts":"1678302046252"} 2023-03-08 19:00:46,252 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-08 19:00:46,252 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:46,252 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for a8e2ccd674e201b22ff6152ea57faf8e: 2023-03-08 19:00:46,253 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=217, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, UNASSIGN in 174 msec 2023-03-08 19:00:46,254 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:46,254 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=a8e2ccd674e201b22ff6152ea57faf8e, regionState=CLOSED 2023-03-08 19:00:46,254 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302046254"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302046254"}]},"ts":"1678302046254"} 2023-03-08 19:00:46,255 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=222, resume processing ppid=218 2023-03-08 19:00:46,255 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=222, ppid=218, state=SUCCESS; CloseRegionProcedure 43584d4ed824c092029e275dd3b82a6c, server=jenkins-hbase6.apache.org,33875,1678302017939 in 172 msec 2023-03-08 19:00:46,255 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=227, resume processing ppid=216 2023-03-08 19:00:46,255 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=227, ppid=216, state=SUCCESS; CloseRegionProcedure f7a0b9e431b1811a0203970d08f9a42d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 165 msec 2023-03-08 19:00:46,256 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=218, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, UNASSIGN in 178 msec 2023-03-08 19:00:46,256 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=216, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, UNASSIGN in 178 msec 2023-03-08 19:00:46,257 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=226, resume processing ppid=221 2023-03-08 19:00:46,257 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=226, ppid=221, state=SUCCESS; CloseRegionProcedure a8e2ccd674e201b22ff6152ea57faf8e, server=jenkins-hbase6.apache.org,46129,1678302018069 in 171 msec 2023-03-08 19:00:46,258 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=221, resume processing ppid=215 2023-03-08 19:00:46,258 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=221, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, UNASSIGN in 180 msec 2023-03-08 19:00:46,259 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302046259"}]},"ts":"1678302046259"} 2023-03-08 19:00:46,260 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1678302045295, state=DISABLED in hbase:meta 2023-03-08 19:00:46,261 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1678302045295 to state=DISABLED 2023-03-08 19:00:46,263 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=215, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1678302045295 in 192 msec 2023-03-08 19:00:46,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-03-08 19:00:46,324 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1678302045295, procId: 215 completed 2023-03-08 19:00:46,325 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1580): Client=jenkins//172.31.2.20 snapshot request for:{ ss=emptySnaptb-1678302045295 table=testCorruptedSnapshot-1678302045295 type=FLUSH ttl=0 } 2023-03-08 19:00:46,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1678302046325 (current time:1678302046325). 2023-03-08 19:00:46,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-03-08 19:00:46,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] zookeeper.ReadOnlyZKClient(139): Connect 0x0dcc8f56 to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:46,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b9ad646, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:46,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:46,336 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:42564, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:46,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0dcc8f56 to 127.0.0.1:59169 2023-03-08 19:00:46,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:46,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-03-08 19:00:46,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1678302045295 table=testCorruptedSnapshot-1678302045295 type=FLUSH ttl=0 } 2023-03-08 19:00:46,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1678302045295, type=EXCLUSIVE 2023-03-08 19:00:46,340 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1678302045295, type=EXCLUSIVE 2023-03-08 19:00:46,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1678302045295 table=testCorruptedSnapshot-1678302045295 type=FLUSH ttl=0 } 2023-03-08 19:00:46,341 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1678302045295 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1678302045295 2023-03-08 19:00:46,341 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=228, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1678302045295, type=EXCLUSIVE; timeout=600000, timestamp=1678302646341 2023-03-08 19:00:46,341 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1678302045295 table=testCorruptedSnapshot-1678302045295 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-03-08 19:00:46,341 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-03-08 19:00:46,357 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1678302045295 table=testCorruptedSnapshot-1678302045295 type=DISABLED ttl=0 } 2023-03-08 19:00:46,359 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:46,359 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:46,359 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:46,359 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:46,359 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:46,359 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:46,360 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:46,360 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:46,361 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:46,361 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:46,362 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:46,362 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:46,362 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:46,362 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:46,364 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:46,364 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:46,364 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:46,365 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:46,366 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:46,366 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:46,367 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-03-08 19:00:46,404 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1678302045295 table=testCorruptedSnapshot-1678302045295 type=DISABLED ttl=0 } as finished. 2023-03-08 19:00:46,404 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1678302045295 2023-03-08 19:00:46,405 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/emptySnaptb-1678302045295 2023-03-08 19:00:46,420 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/emptySnaptb-1678302045295 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/emptySnaptb-1678302045295 2023-03-08 19:00:46,437 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1678302045295 of table testCorruptedSnapshot-1678302045295 completed 2023-03-08 19:00:46,437 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/emptySnaptb-1678302045295 2023-03-08 19:00:46,438 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/emptySnaptb-1678302045295 2023-03-08 19:00:46,438 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1678302045295 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1678302045295 at 1678302046341Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1678302045295 table=testCorruptedSnapshot-1678302045295 type=DISABLED ttl=0 } at 1678302046357 (+16 ms)Consolidate snapshot: emptySnaptb-1678302045295 at 1678302046404 (+47 ms)Loading Region manifests for emptySnaptb-1678302045295 at 1678302046404Writing data manifest for emptySnaptb-1678302045295 at 1678302046408 (+4 ms)Verifying snapshot: emptySnaptb-1678302045295 at 1678302046415 (+7 ms)Snapshot emptySnaptb-1678302045295 of table testCorruptedSnapshot-1678302045295 completed at 1678302046437 (+22 ms) 2023-03-08 19:00:46,439 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=228, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1678302045295, type=EXCLUSIVE 2023-03-08 19:00:46,441 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=228, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1678302045295, type=EXCLUSIVE in 101 msec 2023-03-08 19:00:46,533 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-08 19:00:46,592 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-03-08 19:00:46,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1678302045295 table=testCorruptedSnapshot-1678302045295 type=FLUSH ttl=0 } is done 2023-03-08 19:00:46,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1678302045295 table=testCorruptedSnapshot-1678302045295 type=FLUSH ttl=0 }' has completed, notifying client. 2023-03-08 19:00:46,593 INFO [Listener at localhost/45021] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1678302045295 2023-03-08 19:00:46,594 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$10(2385): Client=jenkins//172.31.2.20 enable testCorruptedSnapshot-1678302045295 2023-03-08 19:00:46,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=229, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1678302045295 2023-03-08 19:00:46,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-03-08 19:00:46,597 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1678302045295 2023-03-08 19:00:46,597 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302046597"}]},"ts":"1678302046597"} 2023-03-08 19:00:46,598 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1678302045295, state=ENABLING in hbase:meta 2023-03-08 19:00:46,600 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-03-08 19:00:46,600 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, ASSIGN}, {pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, ASSIGN}, {pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, ASSIGN}, {pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, ASSIGN}, {pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, ASSIGN}, {pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, ASSIGN}] 2023-03-08 19:00:46,601 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, ASSIGN 2023-03-08 19:00:46,601 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, ASSIGN 2023-03-08 19:00:46,601 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, ASSIGN 2023-03-08 19:00:46,601 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, ASSIGN 2023-03-08 19:00:46,602 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, ASSIGN 2023-03-08 19:00:46,602 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:46,602 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:46,602 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,41161,1678302018011; forceNewPlan=false, retain=true 2023-03-08 19:00:46,602 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:46,602 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:46,602 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,46129,1678302018069; forceNewPlan=false, retain=true 2023-03-08 19:00:46,602 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,41161,1678302018011; forceNewPlan=false, retain=true 2023-03-08 19:00:46,603 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:46,602 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,46129,1678302018069; forceNewPlan=false, retain=true 2023-03-08 19:00:46,603 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, ASSIGN 2023-03-08 19:00:46,603 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,33875,1678302017939; forceNewPlan=false, retain=true 2023-03-08 19:00:46,603 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:46,604 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,33875,1678302017939; forceNewPlan=false, retain=true 2023-03-08 19:00:46,752 INFO [jenkins-hbase6:43879] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-03-08 19:00:46,753 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=a8e2ccd674e201b22ff6152ea57faf8e, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:46,753 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=4446f78ec8b0f1645c8eb2d6d276d62d, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:46,753 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302046753"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302046753"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302046753"}]},"ts":"1678302046753"} 2023-03-08 19:00:46,753 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=f7a0b9e431b1811a0203970d08f9a42d, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:46,753 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=f98146aa6eba5589f1036e30131dab62, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:46,753 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=43584d4ed824c092029e275dd3b82a6c, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:46,753 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302046752"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302046752"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302046752"}]},"ts":"1678302046752"} 2023-03-08 19:00:46,753 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302046753"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302046753"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302046753"}]},"ts":"1678302046753"} 2023-03-08 19:00:46,753 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302046753"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302046753"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302046753"}]},"ts":"1678302046753"} 2023-03-08 19:00:46,753 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302046753"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302046753"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302046753"}]},"ts":"1678302046753"} 2023-03-08 19:00:46,754 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=236, ppid=235, state=RUNNABLE; OpenRegionProcedure a8e2ccd674e201b22ff6152ea57faf8e, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:46,755 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=237, ppid=231, state=RUNNABLE; OpenRegionProcedure f98146aa6eba5589f1036e30131dab62, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:46,755 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=238, ppid=232, state=RUNNABLE; OpenRegionProcedure 43584d4ed824c092029e275dd3b82a6c, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:46,756 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=239, ppid=230, state=RUNNABLE; OpenRegionProcedure f7a0b9e431b1811a0203970d08f9a42d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:46,757 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=240, ppid=234, state=RUNNABLE; OpenRegionProcedure 4446f78ec8b0f1645c8eb2d6d276d62d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:46,757 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=5b3ed2989f5355f239709c8420b1a06c, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:46,757 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302046757"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302046757"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302046757"}]},"ts":"1678302046757"} 2023-03-08 19:00:46,758 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=241, ppid=233, state=RUNNABLE; OpenRegionProcedure 5b3ed2989f5355f239709c8420b1a06c, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:46,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-03-08 19:00:46,908 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:46,908 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:46,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 43584d4ed824c092029e275dd3b82a6c, NAME => 'testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.', STARTKEY => '2', ENDKEY => '3'} 2023-03-08 19:00:46,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a8e2ccd674e201b22ff6152ea57faf8e, NAME => 'testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.', STARTKEY => '5', ENDKEY => ''} 2023-03-08 19:00:46,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:46,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:46,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:46,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:46,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:46,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:46,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:46,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:46,909 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:46,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f7a0b9e431b1811a0203970d08f9a42d, NAME => 'testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.', STARTKEY => '', ENDKEY => '1'} 2023-03-08 19:00:46,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:46,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:46,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:46,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:46,910 INFO [StoreOpener-43584d4ed824c092029e275dd3b82a6c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:46,910 INFO [StoreOpener-a8e2ccd674e201b22ff6152ea57faf8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:46,910 INFO [StoreOpener-f7a0b9e431b1811a0203970d08f9a42d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:46,911 DEBUG [StoreOpener-43584d4ed824c092029e275dd3b82a6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf 2023-03-08 19:00:46,911 DEBUG [StoreOpener-43584d4ed824c092029e275dd3b82a6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf 2023-03-08 19:00:46,911 DEBUG [StoreOpener-a8e2ccd674e201b22ff6152ea57faf8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf 2023-03-08 19:00:46,911 DEBUG [StoreOpener-a8e2ccd674e201b22ff6152ea57faf8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf 2023-03-08 19:00:46,911 INFO [StoreOpener-43584d4ed824c092029e275dd3b82a6c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 43584d4ed824c092029e275dd3b82a6c columnFamilyName cf 2023-03-08 19:00:46,911 DEBUG [StoreOpener-f7a0b9e431b1811a0203970d08f9a42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf 2023-03-08 19:00:46,911 DEBUG [StoreOpener-f7a0b9e431b1811a0203970d08f9a42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf 2023-03-08 19:00:46,911 INFO [StoreOpener-a8e2ccd674e201b22ff6152ea57faf8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a8e2ccd674e201b22ff6152ea57faf8e columnFamilyName cf 2023-03-08 19:00:46,912 INFO [StoreOpener-f7a0b9e431b1811a0203970d08f9a42d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f7a0b9e431b1811a0203970d08f9a42d columnFamilyName cf 2023-03-08 19:00:46,912 INFO [StoreOpener-43584d4ed824c092029e275dd3b82a6c-1] regionserver.HStore(310): Store=43584d4ed824c092029e275dd3b82a6c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:46,912 INFO [StoreOpener-a8e2ccd674e201b22ff6152ea57faf8e-1] regionserver.HStore(310): Store=a8e2ccd674e201b22ff6152ea57faf8e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:46,912 INFO [StoreOpener-f7a0b9e431b1811a0203970d08f9a42d-1] regionserver.HStore(310): Store=f7a0b9e431b1811a0203970d08f9a42d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:46,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:46,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:46,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:46,914 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:46,914 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:46,914 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:46,916 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:46,916 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:46,917 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:46,917 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 43584d4ed824c092029e275dd3b82a6c; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66980102, jitterRate=-0.001918703317642212}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:46,917 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened a8e2ccd674e201b22ff6152ea57faf8e; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72639134, jitterRate=0.08240744471549988}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:46,917 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened f7a0b9e431b1811a0203970d08f9a42d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60764622, jitterRate=-0.09453657269477844}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:46,917 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for a8e2ccd674e201b22ff6152ea57faf8e: 2023-03-08 19:00:46,917 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 43584d4ed824c092029e275dd3b82a6c: 2023-03-08 19:00:46,917 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for f7a0b9e431b1811a0203970d08f9a42d: 2023-03-08 19:00:46,918 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d., pid=239, masterSystemTime=1678302046907 2023-03-08 19:00:46,918 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e., pid=236, masterSystemTime=1678302046906 2023-03-08 19:00:46,918 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c., pid=238, masterSystemTime=1678302046906 2023-03-08 19:00:46,919 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:46,919 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:46,919 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:46,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4446f78ec8b0f1645c8eb2d6d276d62d, NAME => 'testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.', STARTKEY => '4', ENDKEY => '5'} 2023-03-08 19:00:46,920 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=f7a0b9e431b1811a0203970d08f9a42d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:46,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:46,920 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302046920"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302046920"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302046920"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302046920"}]},"ts":"1678302046920"} 2023-03-08 19:00:46,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:46,920 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:46,920 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:46,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f98146aa6eba5589f1036e30131dab62, NAME => 'testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.', STARTKEY => '1', ENDKEY => '2'} 2023-03-08 19:00:46,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:46,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:46,921 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=a8e2ccd674e201b22ff6152ea57faf8e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:46,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:46,921 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302046921"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302046921"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302046921"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302046921"}]},"ts":"1678302046921"} 2023-03-08 19:00:46,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:46,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:46,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:46,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:46,921 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=43584d4ed824c092029e275dd3b82a6c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:46,921 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302046921"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302046921"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302046921"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302046921"}]},"ts":"1678302046921"} 2023-03-08 19:00:46,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:46,922 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:46,922 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:46,922 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5b3ed2989f5355f239709c8420b1a06c, NAME => 'testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.', STARTKEY => '3', ENDKEY => '4'} 2023-03-08 19:00:46,922 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:46,922 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:46,922 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:46,922 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:46,923 INFO [StoreOpener-4446f78ec8b0f1645c8eb2d6d276d62d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:46,923 INFO [StoreOpener-f98146aa6eba5589f1036e30131dab62-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:46,923 INFO [StoreOpener-5b3ed2989f5355f239709c8420b1a06c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:46,924 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=239, resume processing ppid=230 2023-03-08 19:00:46,924 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=239, ppid=230, state=SUCCESS; OpenRegionProcedure f7a0b9e431b1811a0203970d08f9a42d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 165 msec 2023-03-08 19:00:46,924 DEBUG [StoreOpener-4446f78ec8b0f1645c8eb2d6d276d62d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf 2023-03-08 19:00:46,924 DEBUG [StoreOpener-4446f78ec8b0f1645c8eb2d6d276d62d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf 2023-03-08 19:00:46,924 DEBUG [StoreOpener-f98146aa6eba5589f1036e30131dab62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf 2023-03-08 19:00:46,924 DEBUG [StoreOpener-f98146aa6eba5589f1036e30131dab62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf 2023-03-08 19:00:46,924 INFO [StoreOpener-4446f78ec8b0f1645c8eb2d6d276d62d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4446f78ec8b0f1645c8eb2d6d276d62d columnFamilyName cf 2023-03-08 19:00:46,925 INFO [StoreOpener-f98146aa6eba5589f1036e30131dab62-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f98146aa6eba5589f1036e30131dab62 columnFamilyName cf 2023-03-08 19:00:46,925 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=236, resume processing ppid=235 2023-03-08 19:00:46,925 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=236, ppid=235, state=SUCCESS; OpenRegionProcedure a8e2ccd674e201b22ff6152ea57faf8e, server=jenkins-hbase6.apache.org,46129,1678302018069 in 168 msec 2023-03-08 19:00:46,925 DEBUG [StoreOpener-5b3ed2989f5355f239709c8420b1a06c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf 2023-03-08 19:00:46,925 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=230, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, ASSIGN in 324 msec 2023-03-08 19:00:46,925 DEBUG [StoreOpener-5b3ed2989f5355f239709c8420b1a06c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf 2023-03-08 19:00:46,925 INFO [StoreOpener-f98146aa6eba5589f1036e30131dab62-1] regionserver.HStore(310): Store=f98146aa6eba5589f1036e30131dab62/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:46,925 INFO [StoreOpener-4446f78ec8b0f1645c8eb2d6d276d62d-1] regionserver.HStore(310): Store=4446f78ec8b0f1645c8eb2d6d276d62d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:46,926 INFO [StoreOpener-5b3ed2989f5355f239709c8420b1a06c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5b3ed2989f5355f239709c8420b1a06c columnFamilyName cf 2023-03-08 19:00:46,927 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=238, resume processing ppid=232 2023-03-08 19:00:46,927 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:46,927 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:46,927 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=235, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, ASSIGN in 325 msec 2023-03-08 19:00:46,927 INFO [StoreOpener-5b3ed2989f5355f239709c8420b1a06c-1] regionserver.HStore(310): Store=5b3ed2989f5355f239709c8420b1a06c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:46,927 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=238, ppid=232, state=SUCCESS; OpenRegionProcedure 43584d4ed824c092029e275dd3b82a6c, server=jenkins-hbase6.apache.org,33875,1678302017939 in 168 msec 2023-03-08 19:00:46,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:46,928 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=232, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, ASSIGN in 327 msec 2023-03-08 19:00:46,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:46,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:46,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:46,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:46,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:46,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:46,931 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened f98146aa6eba5589f1036e30131dab62; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61033606, jitterRate=-0.09052839875221252}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:46,931 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 4446f78ec8b0f1645c8eb2d6d276d62d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66476567, jitterRate=-0.009421959519386292}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:46,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for f98146aa6eba5589f1036e30131dab62: 2023-03-08 19:00:46,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 4446f78ec8b0f1645c8eb2d6d276d62d: 2023-03-08 19:00:46,932 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d., pid=240, masterSystemTime=1678302046907 2023-03-08 19:00:46,932 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62., pid=237, masterSystemTime=1678302046906 2023-03-08 19:00:46,932 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 5b3ed2989f5355f239709c8420b1a06c; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62589759, jitterRate=-0.06733991205692291}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:46,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 5b3ed2989f5355f239709c8420b1a06c: 2023-03-08 19:00:46,932 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c., pid=241, masterSystemTime=1678302046906 2023-03-08 19:00:46,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:46,933 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:46,933 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=4446f78ec8b0f1645c8eb2d6d276d62d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:46,933 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302046933"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302046933"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302046933"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302046933"}]},"ts":"1678302046933"} 2023-03-08 19:00:46,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:46,934 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:46,934 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=f98146aa6eba5589f1036e30131dab62, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:46,934 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302046934"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302046934"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302046934"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302046934"}]},"ts":"1678302046934"} 2023-03-08 19:00:46,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:46,934 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:46,935 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=5b3ed2989f5355f239709c8420b1a06c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:46,935 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302046935"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302046935"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302046935"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302046935"}]},"ts":"1678302046935"} 2023-03-08 19:00:46,937 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=240, resume processing ppid=234 2023-03-08 19:00:46,937 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=240, ppid=234, state=SUCCESS; OpenRegionProcedure 4446f78ec8b0f1645c8eb2d6d276d62d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 178 msec 2023-03-08 19:00:46,937 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=237, resume processing ppid=231 2023-03-08 19:00:46,937 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=237, ppid=231, state=SUCCESS; OpenRegionProcedure f98146aa6eba5589f1036e30131dab62, server=jenkins-hbase6.apache.org,46129,1678302018069 in 181 msec 2023-03-08 19:00:46,938 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=241, resume processing ppid=233 2023-03-08 19:00:46,938 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=234, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, ASSIGN in 337 msec 2023-03-08 19:00:46,938 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=241, ppid=233, state=SUCCESS; OpenRegionProcedure 5b3ed2989f5355f239709c8420b1a06c, server=jenkins-hbase6.apache.org,33875,1678302017939 in 178 msec 2023-03-08 19:00:46,939 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=231, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, ASSIGN in 337 msec 2023-03-08 19:00:46,939 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=233, resume processing ppid=229 2023-03-08 19:00:46,939 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=233, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, ASSIGN in 338 msec 2023-03-08 19:00:46,940 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302046940"}]},"ts":"1678302046940"} 2023-03-08 19:00:46,941 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1678302045295, state=ENABLED in hbase:meta 2023-03-08 19:00:46,942 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1678302045295' was successfully enabled. 2023-03-08 19:00:46,944 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=229, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1678302045295 in 349 msec 2023-03-08 19:00:47,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-03-08 19:00:47,348 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1678302045295, procId: 229 completed 2023-03-08 19:00:47,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HRegion(8805): writing data to region testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:47,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HRegion(8805): writing data to region testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:47,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HRegion(8805): writing data to region testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:47,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HRegion(8805): writing data to region testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:47,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HRegion(8805): writing data to region testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:47,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HRegion(8805): writing data to region testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:47,364 DEBUG [Listener at localhost/45021] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1678302045295 2023-03-08 19:00:47,364 DEBUG [Listener at localhost/45021] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:47,364 INFO [Listener at localhost/45021] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-03-08 19:00:47,375 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-03-08 19:00:47,376 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-03-08 19:00:47,377 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-03-08 19:00:47,378 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-03-08 19:00:47,379 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-03-08 19:00:47,382 INFO [Listener at localhost/45021] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1678302045295 2023-03-08 19:00:47,382 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$11(2418): Client=jenkins//172.31.2.20 disable testCorruptedSnapshot-1678302045295 2023-03-08 19:00:47,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=242, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1678302045295 2023-03-08 19:00:47,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-03-08 19:00:47,386 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302047386"}]},"ts":"1678302047386"} 2023-03-08 19:00:47,387 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1678302045295, state=DISABLING in hbase:meta 2023-03-08 19:00:47,390 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1678302045295 to state=DISABLING 2023-03-08 19:00:47,390 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, UNASSIGN}, {pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, UNASSIGN}, {pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, UNASSIGN}, {pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, UNASSIGN}, {pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, UNASSIGN}, {pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, UNASSIGN}] 2023-03-08 19:00:47,391 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, UNASSIGN 2023-03-08 19:00:47,391 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, UNASSIGN 2023-03-08 19:00:47,391 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, UNASSIGN 2023-03-08 19:00:47,391 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, UNASSIGN 2023-03-08 19:00:47,391 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, UNASSIGN 2023-03-08 19:00:47,392 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=43584d4ed824c092029e275dd3b82a6c, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:47,392 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=f7a0b9e431b1811a0203970d08f9a42d, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:47,392 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302047392"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302047392"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302047392"}]},"ts":"1678302047392"} 2023-03-08 19:00:47,392 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=5b3ed2989f5355f239709c8420b1a06c, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:47,392 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=f98146aa6eba5589f1036e30131dab62, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:47,392 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302047392"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302047392"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302047392"}]},"ts":"1678302047392"} 2023-03-08 19:00:47,392 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302047392"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302047392"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302047392"}]},"ts":"1678302047392"} 2023-03-08 19:00:47,392 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=a8e2ccd674e201b22ff6152ea57faf8e, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:47,392 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302047392"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302047392"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302047392"}]},"ts":"1678302047392"} 2023-03-08 19:00:47,392 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302047392"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302047392"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302047392"}]},"ts":"1678302047392"} 2023-03-08 19:00:47,393 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=249, ppid=245, state=RUNNABLE; CloseRegionProcedure 43584d4ed824c092029e275dd3b82a6c, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:47,393 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=250, ppid=246, state=RUNNABLE; CloseRegionProcedure 5b3ed2989f5355f239709c8420b1a06c, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:47,394 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=251, ppid=244, state=RUNNABLE; CloseRegionProcedure f98146aa6eba5589f1036e30131dab62, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:47,395 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=252, ppid=243, state=RUNNABLE; CloseRegionProcedure f7a0b9e431b1811a0203970d08f9a42d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:47,395 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=253, ppid=248, state=RUNNABLE; CloseRegionProcedure a8e2ccd674e201b22ff6152ea57faf8e, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:47,395 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, UNASSIGN 2023-03-08 19:00:47,396 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=4446f78ec8b0f1645c8eb2d6d276d62d, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:47,396 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302047396"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302047396"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302047396"}]},"ts":"1678302047396"} 2023-03-08 19:00:47,397 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=254, ppid=247, state=RUNNABLE; CloseRegionProcedure 4446f78ec8b0f1645c8eb2d6d276d62d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:47,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:47,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 43584d4ed824c092029e275dd3b82a6c, disabling compactions & flushes 2023-03-08 19:00:47,545 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:47,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:47,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. after waiting 0 ms 2023-03-08 19:00:47,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:47,545 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 43584d4ed824c092029e275dd3b82a6c 1/1 column families, dataSize=2.42 KB heapSize=5.45 KB 2023-03-08 19:00:47,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:47,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing f98146aa6eba5589f1036e30131dab62, disabling compactions & flushes 2023-03-08 19:00:47,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:47,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:47,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. after waiting 0 ms 2023-03-08 19:00:47,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:47,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing f98146aa6eba5589f1036e30131dab62 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-03-08 19:00:47,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:47,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 4446f78ec8b0f1645c8eb2d6d276d62d, disabling compactions & flushes 2023-03-08 19:00:47,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:47,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:47,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. after waiting 0 ms 2023-03-08 19:00:47,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:47,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 4446f78ec8b0f1645c8eb2d6d276d62d 1/1 column families, dataSize=2.81 KB heapSize=6.30 KB 2023-03-08 19:00:47,567 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202303086b784c2b998442299665458931bcbe8c to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/c4ca4238a0b923820dcc509a6f75849b202303086b784c2b998442299665458931bcbe8c 2023-03-08 19:00:47,569 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202303085e0db7ce3766480792b5825ff3c6ebb6 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/c81e728d9d4c2f636f067f89cc14862c202303085e0db7ce3766480792b5825ff3c6ebb6 2023-03-08 19:00:47,576 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202303081bb21e28a232491abb1c9d6c1f31a8f6 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/a87ff679a2f3e71d9181a67b7542122c202303081bb21e28a232491abb1c9d6c1f31a8f6 2023-03-08 19:00:47,584 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/.tmp/cf/2b777687c347448d9bce41a3a54b997d 2023-03-08 19:00:47,585 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/.tmp/cf/11aca6c127384ce281223ff929d67805 2023-03-08 19:00:47,587 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/.tmp/cf/2dbc45070fca472eb540554d704c241f 2023-03-08 19:00:47,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/.tmp/cf/2b777687c347448d9bce41a3a54b997d as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf/2b777687c347448d9bce41a3a54b997d 2023-03-08 19:00:47,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/.tmp/cf/11aca6c127384ce281223ff929d67805 as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf/11aca6c127384ce281223ff929d67805 2023-03-08 19:00:47,590 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/.tmp/cf/2dbc45070fca472eb540554d704c241f as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf/2dbc45070fca472eb540554d704c241f 2023-03-08 19:00:47,591 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf/2b777687c347448d9bce41a3a54b997d, entries=37, sequenceid=8, filesize=11.3 K 2023-03-08 19:00:47,591 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf/11aca6c127384ce281223ff929d67805, entries=29, sequenceid=8, filesize=9.9 K 2023-03-08 19:00:47,592 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for 43584d4ed824c092029e275dd3b82a6c in 47ms, sequenceid=8, compaction requested=false 2023-03-08 19:00:47,592 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for f98146aa6eba5589f1036e30131dab62 in 46ms, sequenceid=8, compaction requested=false 2023-03-08 19:00:47,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1678302045295' 2023-03-08 19:00:47,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1678302045295' 2023-03-08 19:00:47,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf/2dbc45070fca472eb540554d704c241f, entries=43, sequenceid=8, filesize=12.3 K 2023-03-08 19:00:47,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.81 KB/2879, heapSize ~6.28 KB/6432, currentSize=0 B/0 for 4446f78ec8b0f1645c8eb2d6d276d62d in 47ms, sequenceid=8, compaction requested=false 2023-03-08 19:00:47,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-03-08 19:00:47,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-03-08 19:00:47,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-03-08 19:00:47,597 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:47,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for f98146aa6eba5589f1036e30131dab62: 2023-03-08 19:00:47,597 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:47,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 43584d4ed824c092029e275dd3b82a6c: 2023-03-08 19:00:47,598 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:47,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 4446f78ec8b0f1645c8eb2d6d276d62d: 2023-03-08 19:00:47,599 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:47,599 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:47,599 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing a8e2ccd674e201b22ff6152ea57faf8e, disabling compactions & flushes 2023-03-08 19:00:47,599 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:47,599 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:47,599 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. after waiting 0 ms 2023-03-08 19:00:47,599 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:47,599 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing a8e2ccd674e201b22ff6152ea57faf8e 1/1 column families, dataSize=21.00 KB heapSize=45.39 KB 2023-03-08 19:00:47,600 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=f98146aa6eba5589f1036e30131dab62, regionState=CLOSED 2023-03-08 19:00:47,600 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302047600"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302047600"}]},"ts":"1678302047600"} 2023-03-08 19:00:47,600 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:47,600 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:47,600 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 5b3ed2989f5355f239709c8420b1a06c, disabling compactions & flushes 2023-03-08 19:00:47,600 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:47,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:47,601 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=43584d4ed824c092029e275dd3b82a6c, regionState=CLOSED 2023-03-08 19:00:47,601 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302047601"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302047601"}]},"ts":"1678302047601"} 2023-03-08 19:00:47,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. after waiting 0 ms 2023-03-08 19:00:47,601 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:47,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:47,601 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 5b3ed2989f5355f239709c8420b1a06c 1/1 column families, dataSize=2.68 KB heapSize=6.02 KB 2023-03-08 19:00:47,601 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:47,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing f7a0b9e431b1811a0203970d08f9a42d, disabling compactions & flushes 2023-03-08 19:00:47,601 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:47,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:47,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. after waiting 0 ms 2023-03-08 19:00:47,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:47,601 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing f7a0b9e431b1811a0203970d08f9a42d 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-03-08 19:00:47,601 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=4446f78ec8b0f1645c8eb2d6d276d62d, regionState=CLOSED 2023-03-08 19:00:47,602 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302047601"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302047601"}]},"ts":"1678302047601"} 2023-03-08 19:00:47,604 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=251, resume processing ppid=244 2023-03-08 19:00:47,604 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=251, ppid=244, state=SUCCESS; CloseRegionProcedure f98146aa6eba5589f1036e30131dab62, server=jenkins-hbase6.apache.org,46129,1678302018069 in 208 msec 2023-03-08 19:00:47,605 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=249, resume processing ppid=245 2023-03-08 19:00:47,605 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=249, ppid=245, state=SUCCESS; CloseRegionProcedure 43584d4ed824c092029e275dd3b82a6c, server=jenkins-hbase6.apache.org,33875,1678302017939 in 209 msec 2023-03-08 19:00:47,605 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=254, resume processing ppid=247 2023-03-08 19:00:47,606 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=244, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, UNASSIGN in 214 msec 2023-03-08 19:00:47,606 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=254, ppid=247, state=SUCCESS; CloseRegionProcedure 4446f78ec8b0f1645c8eb2d6d276d62d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 206 msec 2023-03-08 19:00:47,606 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=245, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, UNASSIGN in 215 msec 2023-03-08 19:00:47,607 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=247, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, UNASSIGN in 216 msec 2023-03-08 19:00:47,632 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023030844f14ff157f64ee1a242d8ff2eb93263 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/d41d8cd98f00b204e9800998ecf8427e2023030844f14ff157f64ee1a242d8ff2eb93263 2023-03-08 19:00:47,634 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230308224d67a9f09e41b7b978a985bb54751e to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/e4da3b7fbbce2345d7772b0674a318d520230308224d67a9f09e41b7b978a985bb54751e 2023-03-08 19:00:47,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-03-08 19:00:47,642 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/.tmp/cf/bb5840f0dc9a4ec99bcb3ce4b4e83dee 2023-03-08 19:00:47,645 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=21.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/.tmp/cf/9922a96e482943fdb71ab70b22b06658 2023-03-08 19:00:47,645 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/.tmp/cf/bb5840f0dc9a4ec99bcb3ce4b4e83dee as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf/bb5840f0dc9a4ec99bcb3ce4b4e83dee 2023-03-08 19:00:47,648 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/.tmp/cf/9922a96e482943fdb71ab70b22b06658 as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf/9922a96e482943fdb71ab70b22b06658 2023-03-08 19:00:47,648 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf/bb5840f0dc9a4ec99bcb3ce4b4e83dee, entries=29, sequenceid=8, filesize=9.9 K 2023-03-08 19:00:47,649 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for f7a0b9e431b1811a0203970d08f9a42d in 48ms, sequenceid=8, compaction requested=false 2023-03-08 19:00:47,650 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf/9922a96e482943fdb71ab70b22b06658, entries=321, sequenceid=8, filesize=60.3 K 2023-03-08 19:00:47,651 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~21.00 KB/21503, heapSize ~45.38 KB/46464, currentSize=0 B/0 for a8e2ccd674e201b22ff6152ea57faf8e in 52ms, sequenceid=8, compaction requested=false 2023-03-08 19:00:47,652 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-03-08 19:00:47,652 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:47,652 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for f7a0b9e431b1811a0203970d08f9a42d: 2023-03-08 19:00:47,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:47,654 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=f7a0b9e431b1811a0203970d08f9a42d, regionState=CLOSED 2023-03-08 19:00:47,654 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-03-08 19:00:47,654 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302047654"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302047654"}]},"ts":"1678302047654"} 2023-03-08 19:00:47,654 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:47,654 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for a8e2ccd674e201b22ff6152ea57faf8e: 2023-03-08 19:00:47,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:47,656 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=a8e2ccd674e201b22ff6152ea57faf8e, regionState=CLOSED 2023-03-08 19:00:47,656 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302047656"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302047656"}]},"ts":"1678302047656"} 2023-03-08 19:00:47,657 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=252, resume processing ppid=243 2023-03-08 19:00:47,657 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=252, ppid=243, state=SUCCESS; CloseRegionProcedure f7a0b9e431b1811a0203970d08f9a42d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 261 msec 2023-03-08 19:00:47,658 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=243, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, UNASSIGN in 267 msec 2023-03-08 19:00:47,658 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=253, resume processing ppid=248 2023-03-08 19:00:47,658 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=253, ppid=248, state=SUCCESS; CloseRegionProcedure a8e2ccd674e201b22ff6152ea57faf8e, server=jenkins-hbase6.apache.org,46129,1678302018069 in 262 msec 2023-03-08 19:00:47,659 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=248, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, UNASSIGN in 268 msec 2023-03-08 19:00:48,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230308e93000cb1d43425da1d8a7cf3446f228 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308e93000cb1d43425da1d8a7cf3446f228 2023-03-08 19:00:48,040 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/.tmp/cf/152f69f176104902b666b5d244c7544a 2023-03-08 19:00:48,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/.tmp/cf/152f69f176104902b666b5d244c7544a as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf/152f69f176104902b666b5d244c7544a 2023-03-08 19:00:48,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf/152f69f176104902b666b5d244c7544a, entries=41, sequenceid=8, filesize=12.0 K 2023-03-08 19:00:48,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.68 KB/2745, heapSize ~6 KB/6144, currentSize=0 B/0 for 5b3ed2989f5355f239709c8420b1a06c in 446ms, sequenceid=8, compaction requested=false 2023-03-08 19:00:48,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-03-08 19:00:48,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:48,050 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 5b3ed2989f5355f239709c8420b1a06c: 2023-03-08 19:00:48,051 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:48,051 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=5b3ed2989f5355f239709c8420b1a06c, regionState=CLOSED 2023-03-08 19:00:48,051 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302048051"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302048051"}]},"ts":"1678302048051"} 2023-03-08 19:00:48,053 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=250, resume processing ppid=246 2023-03-08 19:00:48,053 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=250, ppid=246, state=SUCCESS; CloseRegionProcedure 5b3ed2989f5355f239709c8420b1a06c, server=jenkins-hbase6.apache.org,33875,1678302017939 in 659 msec 2023-03-08 19:00:48,055 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=246, resume processing ppid=242 2023-03-08 19:00:48,055 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=246, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, UNASSIGN in 663 msec 2023-03-08 19:00:48,055 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302048055"}]},"ts":"1678302048055"} 2023-03-08 19:00:48,056 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1678302045295, state=DISABLED in hbase:meta 2023-03-08 19:00:48,058 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1678302045295 to state=DISABLED 2023-03-08 19:00:48,059 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=242, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1678302045295 in 675 msec 2023-03-08 19:00:48,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-03-08 19:00:48,138 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1678302045295, procId: 242 completed 2023-03-08 19:00:48,139 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1580): Client=jenkins//172.31.2.20 snapshot request for:{ ss=snaptb0-1678302045295 table=testCorruptedSnapshot-1678302045295 type=FLUSH ttl=0 } 2023-03-08 19:00:48,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1678302048139 (current time:1678302048139). 2023-03-08 19:00:48,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-03-08 19:00:48,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] zookeeper.ReadOnlyZKClient(139): Connect 0x11acd5b4 to 127.0.0.1:59169 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-08 19:00:48,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78735ee8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-08 19:00:48,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-08 19:00:48,148 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:42578, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-08 19:00:48,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x11acd5b4 to 127.0.0.1:59169 2023-03-08 19:00:48,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:48,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-03-08 19:00:48,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1678302045295 table=testCorruptedSnapshot-1678302045295 type=FLUSH ttl=0 } 2023-03-08 19:00:48,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1678302045295, type=EXCLUSIVE 2023-03-08 19:00:48,152 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1678302045295, type=EXCLUSIVE 2023-03-08 19:00:48,152 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=255, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1678302045295, type=EXCLUSIVE; timeout=600000, timestamp=1678302648152 2023-03-08 19:00:48,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1678302045295 table=testCorruptedSnapshot-1678302045295 type=FLUSH ttl=0 } 2023-03-08 19:00:48,152 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1678302045295 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1678302045295 2023-03-08 19:00:48,153 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1678302045295 table=testCorruptedSnapshot-1678302045295 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-03-08 19:00:48,153 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-03-08 19:00:48,162 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1678302045295 table=testCorruptedSnapshot-1678302045295 type=DISABLED ttl=0 } 2023-03-08 19:00:48,164 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:48,164 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:48,164 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:48,164 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:48,164 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:48,164 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:48,164 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:48,164 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:48,164 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:48,164 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:48,164 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:48,164 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:48,164 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-03-08 19:00:48,164 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-03-08 19:00:48,166 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf/2b777687c347448d9bce41a3a54b997d] hfiles 2023-03-08 19:00:48,166 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf/2b777687c347448d9bce41a3a54b997d 2023-03-08 19:00:48,166 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf/bb5840f0dc9a4ec99bcb3ce4b4e83dee] hfiles 2023-03-08 19:00:48,166 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf/bb5840f0dc9a4ec99bcb3ce4b4e83dee 2023-03-08 19:00:48,166 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf/152f69f176104902b666b5d244c7544a] hfiles 2023-03-08 19:00:48,166 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf/152f69f176104902b666b5d244c7544a 2023-03-08 19:00:48,166 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf/2dbc45070fca472eb540554d704c241f] hfiles 2023-03-08 19:00:48,166 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf/9922a96e482943fdb71ab70b22b06658] hfiles 2023-03-08 19:00:48,166 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf/2dbc45070fca472eb540554d704c241f 2023-03-08 19:00:48,166 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf/9922a96e482943fdb71ab70b22b06658 2023-03-08 19:00:48,167 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf/11aca6c127384ce281223ff929d67805] hfiles 2023-03-08 19:00:48,167 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf/11aca6c127384ce281223ff929d67805 2023-03-08 19:00:48,170 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/a87ff679a2f3e71d9181a67b7542122c202303081bb21e28a232491abb1c9d6c1f31a8f6, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/c4ca4238a0b923820dcc509a6f75849b202303086b784c2b998442299665458931bcbe8c, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/c81e728d9d4c2f636f067f89cc14862c202303085e0db7ce3766480792b5825ff3c6ebb6, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/d41d8cd98f00b204e9800998ecf8427e2023030844f14ff157f64ee1a242d8ff2eb93263, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/e4da3b7fbbce2345d7772b0674a318d520230308224d67a9f09e41b7b978a985bb54751e, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308e93000cb1d43425da1d8a7cf3446f228] hfiles 2023-03-08 19:00:48,170 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/a87ff679a2f3e71d9181a67b7542122c202303081bb21e28a232491abb1c9d6c1f31a8f6 2023-03-08 19:00:48,170 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/c4ca4238a0b923820dcc509a6f75849b202303086b784c2b998442299665458931bcbe8c 2023-03-08 19:00:48,170 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/c81e728d9d4c2f636f067f89cc14862c202303085e0db7ce3766480792b5825ff3c6ebb6 2023-03-08 19:00:48,170 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/d41d8cd98f00b204e9800998ecf8427e2023030844f14ff157f64ee1a242d8ff2eb93263 2023-03-08 19:00:48,170 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/e4da3b7fbbce2345d7772b0674a318d520230308224d67a9f09e41b7b978a985bb54751e 2023-03-08 19:00:48,170 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308e93000cb1d43425da1d8a7cf3446f228 2023-03-08 19:00:48,403 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-03-08 19:00:48,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1678302045295 table=testCorruptedSnapshot-1678302045295 type=FLUSH ttl=0 } is done 2023-03-08 19:00:48,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snaptb0-1678302045295 table=testCorruptedSnapshot-1678302045295 type=FLUSH ttl=0 }' is still in progress! 2023-03-08 19:00:48,404 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-03-08 19:00:48,586 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1678302045295 table=testCorruptedSnapshot-1678302045295 type=DISABLED ttl=0 } as finished. 2023-03-08 19:00:48,586 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1678302045295 2023-03-08 19:00:48,587 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/snaptb0-1678302045295 2023-03-08 19:00:48,604 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/snaptb0-1678302045295 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/snaptb0-1678302045295 2023-03-08 19:00:48,626 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1678302045295 of table testCorruptedSnapshot-1678302045295 completed 2023-03-08 19:00:48,626 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/snaptb0-1678302045295 2023-03-08 19:00:48,627 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.hbase-snapshot/.tmp/snaptb0-1678302045295 2023-03-08 19:00:48,627 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1678302045295 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1678302045295 at 1678302048152Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1678302045295 table=testCorruptedSnapshot-1678302045295 type=DISABLED ttl=0 } at 1678302048162 (+10 ms)Consolidate snapshot: snaptb0-1678302045295 at 1678302048586 (+424 ms)Loading Region manifests for snaptb0-1678302045295 at 1678302048586Writing data manifest for snaptb0-1678302045295 at 1678302048590 (+4 ms)Verifying snapshot: snaptb0-1678302045295 at 1678302048597 (+7 ms)Snapshot snaptb0-1678302045295 of table testCorruptedSnapshot-1678302045295 completed at 1678302048626 (+29 ms) 2023-03-08 19:00:48,628 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=255, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1678302045295, type=EXCLUSIVE 2023-03-08 19:00:48,629 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=255, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1678302045295, type=EXCLUSIVE in 478 msec 2023-03-08 19:00:48,904 DEBUG [Listener at localhost/45021] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-03-08 19:00:48,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1678302045295 table=testCorruptedSnapshot-1678302045295 type=FLUSH ttl=0 } is done 2023-03-08 19:00:48,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1678302045295 table=testCorruptedSnapshot-1678302045295 type=FLUSH ttl=0 }' has completed, notifying client. 2023-03-08 19:00:48,905 INFO [Listener at localhost/45021] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1678302045295 2023-03-08 19:00:48,906 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$10(2385): Client=jenkins//172.31.2.20 enable testCorruptedSnapshot-1678302045295 2023-03-08 19:00:48,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=256, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1678302045295 2023-03-08 19:00:48,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-03-08 19:00:48,909 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1678302045295 2023-03-08 19:00:48,909 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302048909"}]},"ts":"1678302048909"} 2023-03-08 19:00:48,910 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1678302045295, state=ENABLING in hbase:meta 2023-03-08 19:00:48,913 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-03-08 19:00:48,913 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, ASSIGN}, {pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, ASSIGN}, {pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, ASSIGN}, {pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, ASSIGN}, {pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, ASSIGN}, {pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, ASSIGN}] 2023-03-08 19:00:48,914 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, ASSIGN 2023-03-08 19:00:48,914 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, ASSIGN 2023-03-08 19:00:48,914 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, ASSIGN 2023-03-08 19:00:48,914 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, ASSIGN 2023-03-08 19:00:48,914 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, ASSIGN 2023-03-08 19:00:48,914 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:48,914 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:48,914 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,46129,1678302018069; forceNewPlan=false, retain=true 2023-03-08 19:00:48,915 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,41161,1678302018011; forceNewPlan=false, retain=true 2023-03-08 19:00:48,914 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:48,914 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:48,915 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,33875,1678302017939; forceNewPlan=false, retain=true 2023-03-08 19:00:48,915 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,33875,1678302017939; forceNewPlan=false, retain=true 2023-03-08 19:00:48,915 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:48,915 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,46129,1678302018069; forceNewPlan=false, retain=true 2023-03-08 19:00:48,915 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, ASSIGN 2023-03-08 19:00:48,916 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:48,916 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, ASSIGN; state=CLOSED, location=jenkins-hbase6.apache.org,41161,1678302018011; forceNewPlan=false, retain=true 2023-03-08 19:00:49,065 INFO [jenkins-hbase6:43879] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-03-08 19:00:49,065 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=4446f78ec8b0f1645c8eb2d6d276d62d, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:49,065 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=f7a0b9e431b1811a0203970d08f9a42d, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:49,065 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=a8e2ccd674e201b22ff6152ea57faf8e, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:49,065 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302049065"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302049065"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302049065"}]},"ts":"1678302049065"} 2023-03-08 19:00:49,065 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302049065"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302049065"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302049065"}]},"ts":"1678302049065"} 2023-03-08 19:00:49,065 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=43584d4ed824c092029e275dd3b82a6c, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:49,065 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=f98146aa6eba5589f1036e30131dab62, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:49,065 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302049065"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302049065"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302049065"}]},"ts":"1678302049065"} 2023-03-08 19:00:49,066 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302049065"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302049065"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302049065"}]},"ts":"1678302049065"} 2023-03-08 19:00:49,066 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302049065"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302049065"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302049065"}]},"ts":"1678302049065"} 2023-03-08 19:00:49,067 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=263, ppid=257, state=RUNNABLE; OpenRegionProcedure f7a0b9e431b1811a0203970d08f9a42d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:49,067 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=264, ppid=262, state=RUNNABLE; OpenRegionProcedure a8e2ccd674e201b22ff6152ea57faf8e, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:49,068 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=265, ppid=261, state=RUNNABLE; OpenRegionProcedure 4446f78ec8b0f1645c8eb2d6d276d62d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:49,069 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=266, ppid=258, state=RUNNABLE; OpenRegionProcedure f98146aa6eba5589f1036e30131dab62, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:49,069 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=267, ppid=259, state=RUNNABLE; OpenRegionProcedure 43584d4ed824c092029e275dd3b82a6c, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:49,069 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=5b3ed2989f5355f239709c8420b1a06c, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:49,070 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302049069"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302049069"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302049069"}]},"ts":"1678302049069"} 2023-03-08 19:00:49,071 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=268, ppid=260, state=RUNNABLE; OpenRegionProcedure 5b3ed2989f5355f239709c8420b1a06c, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:49,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-03-08 19:00:49,220 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:49,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4446f78ec8b0f1645c8eb2d6d276d62d, NAME => 'testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.', STARTKEY => '4', ENDKEY => '5'} 2023-03-08 19:00:49,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:49,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:49,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:49,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:49,221 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:49,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a8e2ccd674e201b22ff6152ea57faf8e, NAME => 'testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.', STARTKEY => '5', ENDKEY => ''} 2023-03-08 19:00:49,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:49,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:49,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:49,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:49,222 INFO [StoreOpener-4446f78ec8b0f1645c8eb2d6d276d62d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:49,223 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:49,223 INFO [StoreOpener-a8e2ccd674e201b22ff6152ea57faf8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:49,223 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5b3ed2989f5355f239709c8420b1a06c, NAME => 'testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.', STARTKEY => '3', ENDKEY => '4'} 2023-03-08 19:00:49,223 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:49,223 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:49,223 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:49,223 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:49,224 DEBUG [StoreOpener-4446f78ec8b0f1645c8eb2d6d276d62d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf 2023-03-08 19:00:49,224 DEBUG [StoreOpener-4446f78ec8b0f1645c8eb2d6d276d62d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf 2023-03-08 19:00:49,224 DEBUG [StoreOpener-a8e2ccd674e201b22ff6152ea57faf8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf 2023-03-08 19:00:49,224 DEBUG [StoreOpener-a8e2ccd674e201b22ff6152ea57faf8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf 2023-03-08 19:00:49,224 INFO [StoreOpener-5b3ed2989f5355f239709c8420b1a06c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:49,224 INFO [StoreOpener-4446f78ec8b0f1645c8eb2d6d276d62d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4446f78ec8b0f1645c8eb2d6d276d62d columnFamilyName cf 2023-03-08 19:00:49,224 INFO [StoreOpener-a8e2ccd674e201b22ff6152ea57faf8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a8e2ccd674e201b22ff6152ea57faf8e columnFamilyName cf 2023-03-08 19:00:49,225 DEBUG [StoreOpener-5b3ed2989f5355f239709c8420b1a06c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf 2023-03-08 19:00:49,225 DEBUG [StoreOpener-5b3ed2989f5355f239709c8420b1a06c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf 2023-03-08 19:00:49,225 INFO [StoreOpener-5b3ed2989f5355f239709c8420b1a06c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5b3ed2989f5355f239709c8420b1a06c columnFamilyName cf 2023-03-08 19:00:49,228 DEBUG [StoreOpener-a8e2ccd674e201b22ff6152ea57faf8e-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf/9922a96e482943fdb71ab70b22b06658 2023-03-08 19:00:49,228 DEBUG [StoreOpener-4446f78ec8b0f1645c8eb2d6d276d62d-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf/2dbc45070fca472eb540554d704c241f 2023-03-08 19:00:49,228 INFO [StoreOpener-a8e2ccd674e201b22ff6152ea57faf8e-1] regionserver.HStore(310): Store=a8e2ccd674e201b22ff6152ea57faf8e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:49,228 INFO [StoreOpener-4446f78ec8b0f1645c8eb2d6d276d62d-1] regionserver.HStore(310): Store=4446f78ec8b0f1645c8eb2d6d276d62d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:49,229 DEBUG [StoreOpener-5b3ed2989f5355f239709c8420b1a06c-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf/152f69f176104902b666b5d244c7544a 2023-03-08 19:00:49,229 INFO [StoreOpener-5b3ed2989f5355f239709c8420b1a06c-1] regionserver.HStore(310): Store=5b3ed2989f5355f239709c8420b1a06c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:49,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:49,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:49,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:49,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:49,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:49,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:49,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:49,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:49,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:49,233 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 4446f78ec8b0f1645c8eb2d6d276d62d; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66870108, jitterRate=-0.003557741641998291}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:49,233 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened a8e2ccd674e201b22ff6152ea57faf8e; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59824433, jitterRate=-0.10854648053646088}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:49,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 4446f78ec8b0f1645c8eb2d6d276d62d: 2023-03-08 19:00:49,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for a8e2ccd674e201b22ff6152ea57faf8e: 2023-03-08 19:00:49,233 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 5b3ed2989f5355f239709c8420b1a06c; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72387569, jitterRate=0.07865883409976959}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:49,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 5b3ed2989f5355f239709c8420b1a06c: 2023-03-08 19:00:49,234 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e., pid=264, masterSystemTime=1678302049219 2023-03-08 19:00:49,234 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c., pid=268, masterSystemTime=1678302049221 2023-03-08 19:00:49,234 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d., pid=265, masterSystemTime=1678302049218 2023-03-08 19:00:49,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:49,235 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:49,235 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:49,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f98146aa6eba5589f1036e30131dab62, NAME => 'testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.', STARTKEY => '1', ENDKEY => '2'} 2023-03-08 19:00:49,235 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=a8e2ccd674e201b22ff6152ea57faf8e, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:49,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:49,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:49,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:49,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:49,235 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302049235"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302049235"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302049235"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302049235"}]},"ts":"1678302049235"} 2023-03-08 19:00:49,236 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=4446f78ec8b0f1645c8eb2d6d276d62d, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:49,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:49,236 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302049236"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302049236"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302049236"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302049236"}]},"ts":"1678302049236"} 2023-03-08 19:00:49,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:49,236 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:49,236 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:49,236 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:49,236 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:49,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 43584d4ed824c092029e275dd3b82a6c, NAME => 'testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.', STARTKEY => '2', ENDKEY => '3'} 2023-03-08 19:00:49,237 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=5b3ed2989f5355f239709c8420b1a06c, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:49,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:49,237 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302049237"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302049237"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302049237"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302049237"}]},"ts":"1678302049237"} 2023-03-08 19:00:49,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f7a0b9e431b1811a0203970d08f9a42d, NAME => 'testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.', STARTKEY => '', ENDKEY => '1'} 2023-03-08 19:00:49,237 INFO [StoreOpener-f98146aa6eba5589f1036e30131dab62-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:49,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:49,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:49,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:49,239 INFO [StoreOpener-43584d4ed824c092029e275dd3b82a6c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:49,239 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=264, resume processing ppid=262 2023-03-08 19:00:49,239 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=264, ppid=262, state=SUCCESS; OpenRegionProcedure a8e2ccd674e201b22ff6152ea57faf8e, server=jenkins-hbase6.apache.org,46129,1678302018069 in 170 msec 2023-03-08 19:00:49,240 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=265, resume processing ppid=261 2023-03-08 19:00:49,240 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=265, ppid=261, state=SUCCESS; OpenRegionProcedure 4446f78ec8b0f1645c8eb2d6d276d62d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 170 msec 2023-03-08 19:00:49,240 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=262, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, ASSIGN in 326 msec 2023-03-08 19:00:49,240 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=268, resume processing ppid=260 2023-03-08 19:00:49,240 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=268, ppid=260, state=SUCCESS; OpenRegionProcedure 5b3ed2989f5355f239709c8420b1a06c, server=jenkins-hbase6.apache.org,33875,1678302017939 in 168 msec 2023-03-08 19:00:49,241 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=261, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, ASSIGN in 327 msec 2023-03-08 19:00:49,241 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=260, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, ASSIGN in 328 msec 2023-03-08 19:00:49,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1678302045295 f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:49,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-08 19:00:49,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:49,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:49,244 INFO [StoreOpener-f7a0b9e431b1811a0203970d08f9a42d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:49,244 DEBUG [StoreOpener-f98146aa6eba5589f1036e30131dab62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf 2023-03-08 19:00:49,244 DEBUG [StoreOpener-43584d4ed824c092029e275dd3b82a6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf 2023-03-08 19:00:49,244 DEBUG [StoreOpener-f98146aa6eba5589f1036e30131dab62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf 2023-03-08 19:00:49,244 DEBUG [StoreOpener-43584d4ed824c092029e275dd3b82a6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf 2023-03-08 19:00:49,244 INFO [StoreOpener-f98146aa6eba5589f1036e30131dab62-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f98146aa6eba5589f1036e30131dab62 columnFamilyName cf 2023-03-08 19:00:49,245 INFO [StoreOpener-43584d4ed824c092029e275dd3b82a6c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 43584d4ed824c092029e275dd3b82a6c columnFamilyName cf 2023-03-08 19:00:49,245 DEBUG [StoreOpener-f7a0b9e431b1811a0203970d08f9a42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf 2023-03-08 19:00:49,245 DEBUG [StoreOpener-f7a0b9e431b1811a0203970d08f9a42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf 2023-03-08 19:00:49,245 INFO [StoreOpener-f7a0b9e431b1811a0203970d08f9a42d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f7a0b9e431b1811a0203970d08f9a42d columnFamilyName cf 2023-03-08 19:00:49,249 DEBUG [StoreOpener-43584d4ed824c092029e275dd3b82a6c-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf/2b777687c347448d9bce41a3a54b997d 2023-03-08 19:00:49,249 DEBUG [StoreOpener-f98146aa6eba5589f1036e30131dab62-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf/11aca6c127384ce281223ff929d67805 2023-03-08 19:00:49,250 INFO [StoreOpener-43584d4ed824c092029e275dd3b82a6c-1] regionserver.HStore(310): Store=43584d4ed824c092029e275dd3b82a6c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:49,250 INFO [StoreOpener-f98146aa6eba5589f1036e30131dab62-1] regionserver.HStore(310): Store=f98146aa6eba5589f1036e30131dab62/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:49,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:49,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:49,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:49,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:49,252 DEBUG [StoreOpener-f7a0b9e431b1811a0203970d08f9a42d-1] regionserver.HStore(539): loaded hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf/bb5840f0dc9a4ec99bcb3ce4b4e83dee 2023-03-08 19:00:49,253 INFO [StoreOpener-f7a0b9e431b1811a0203970d08f9a42d-1] regionserver.HStore(310): Store=f7a0b9e431b1811a0203970d08f9a42d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-08 19:00:49,253 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:49,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:49,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:49,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:49,255 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 43584d4ed824c092029e275dd3b82a6c; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70500168, jitterRate=0.05053436756134033}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:49,255 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 43584d4ed824c092029e275dd3b82a6c: 2023-03-08 19:00:49,255 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened f98146aa6eba5589f1036e30131dab62; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67544590, jitterRate=0.006492823362350464}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:49,255 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for f98146aa6eba5589f1036e30131dab62: 2023-03-08 19:00:49,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62., pid=266, masterSystemTime=1678302049219 2023-03-08 19:00:49,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c., pid=267, masterSystemTime=1678302049221 2023-03-08 19:00:49,257 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:49,257 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:49,257 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:49,257 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=43584d4ed824c092029e275dd3b82a6c, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:49,258 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302049257"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302049257"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302049257"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302049257"}]},"ts":"1678302049257"} 2023-03-08 19:00:49,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:49,258 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:49,258 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened f7a0b9e431b1811a0203970d08f9a42d; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67925758, jitterRate=0.012172669172286987}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-08 19:00:49,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for f7a0b9e431b1811a0203970d08f9a42d: 2023-03-08 19:00:49,258 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=f98146aa6eba5589f1036e30131dab62, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:49,259 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302049258"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302049258"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302049258"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302049258"}]},"ts":"1678302049258"} 2023-03-08 19:00:49,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d., pid=263, masterSystemTime=1678302049218 2023-03-08 19:00:49,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:49,262 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:49,262 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=f7a0b9e431b1811a0203970d08f9a42d, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:49,262 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302049262"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1678302049262"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1678302049262"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1678302049262"}]},"ts":"1678302049262"} 2023-03-08 19:00:49,263 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=267, resume processing ppid=259 2023-03-08 19:00:49,263 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=267, ppid=259, state=SUCCESS; OpenRegionProcedure 43584d4ed824c092029e275dd3b82a6c, server=jenkins-hbase6.apache.org,33875,1678302017939 in 190 msec 2023-03-08 19:00:49,263 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=266, resume processing ppid=258 2023-03-08 19:00:49,263 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=266, ppid=258, state=SUCCESS; OpenRegionProcedure f98146aa6eba5589f1036e30131dab62, server=jenkins-hbase6.apache.org,46129,1678302018069 in 192 msec 2023-03-08 19:00:49,264 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=259, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, ASSIGN in 350 msec 2023-03-08 19:00:49,264 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=258, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, ASSIGN in 350 msec 2023-03-08 19:00:49,265 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=263, resume processing ppid=257 2023-03-08 19:00:49,265 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=263, ppid=257, state=SUCCESS; OpenRegionProcedure f7a0b9e431b1811a0203970d08f9a42d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 197 msec 2023-03-08 19:00:49,266 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=257, resume processing ppid=256 2023-03-08 19:00:49,266 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=257, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, ASSIGN in 352 msec 2023-03-08 19:00:49,267 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302049267"}]},"ts":"1678302049267"} 2023-03-08 19:00:49,268 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1678302045295, state=ENABLED in hbase:meta 2023-03-08 19:00:49,269 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1678302045295' was successfully enabled. 2023-03-08 19:00:49,271 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=256, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1678302045295 in 364 msec 2023-03-08 19:00:49,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-03-08 19:00:49,660 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1678302045295, procId: 256 completed 2023-03-08 19:00:49,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HRegion(8805): writing data to region testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:49,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HRegion(8805): writing data to region testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:49,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33875] regionserver.HRegion(8805): writing data to region testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:49,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41161] regionserver.HRegion(8805): writing data to region testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:49,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HRegion(8805): writing data to region testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:49,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46129] regionserver.HRegion(8805): writing data to region testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. with WAL disabled. Data may be lost in the event of a crash. 2023-03-08 19:00:49,673 DEBUG [Listener at localhost/45021] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1678302045295 2023-03-08 19:00:49,673 DEBUG [Listener at localhost/45021] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:49,674 INFO [Listener at localhost/45021] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-03-08 19:00:49,769 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-03-08 19:00:49,836 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-03-08 19:00:49,917 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-03-08 19:00:50,005 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-03-08 19:00:50,095 DEBUG [Listener at localhost/45021] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-03-08 19:00:50,797 ERROR [VerifySnapshot-pool-0] snapshot.SnapshotReferenceUtil(301): Can't find hfile: a87ff679a2f3e71d9181a67b7542122c202303081bb21e28a232491abb1c9d6c1f31a8f6 in the real (hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/a87ff679a2f3e71d9181a67b7542122c202303081bb21e28a232491abb1c9d6c1f31a8f6) or archive (hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/a87ff679a2f3e71d9181a67b7542122c202303081bb21e28a232491abb1c9d6c1f31a8f6) directory for the primary table. 2023-03-08 19:00:50,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: a87ff679a2f3e71d9181a67b7542122c202303081bb21e28a232491abb1c9d6c1f31a8f6 in the real (hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/a87ff679a2f3e71d9181a67b7542122c202303081bb21e28a232491abb1c9d6c1f31a8f6) or archive (hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/a87ff679a2f3e71d9181a67b7542122c202303081bb21e28a232491abb1c9d6c1f31a8f6) directory for the primary table. at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:225) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:187) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.verifySnapshot(SnapshotReferenceUtil.java:164) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:827) at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2506) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2502) at org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1469) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-08 19:00:50,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] ipc.CallRunner(144): callId: 362 service: MasterService methodName: RestoreSnapshot size: 115 connection: 172.31.2.20:36330 deadline: 1678302110793, exception=org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: a87ff679a2f3e71d9181a67b7542122c202303081bb21e28a232491abb1c9d6c1f31a8f6 in the real (hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/a87ff679a2f3e71d9181a67b7542122c202303081bb21e28a232491abb1c9d6c1f31a8f6) or archive (hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/a87ff679a2f3e71d9181a67b7542122c202303081bb21e28a232491abb1c9d6c1f31a8f6) directory for the primary table. 2023-03-08 19:00:50,802 INFO [Listener at localhost/45021] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1678302045295 2023-03-08 19:00:50,802 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$11(2418): Client=jenkins//172.31.2.20 disable testCorruptedSnapshot-1678302045295 2023-03-08 19:00:50,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=270, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1678302045295 2023-03-08 19:00:50,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-03-08 19:00:50,805 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302050805"}]},"ts":"1678302050805"} 2023-03-08 19:00:50,806 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1678302045295, state=DISABLING in hbase:meta 2023-03-08 19:00:50,807 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1678302045295 to state=DISABLING 2023-03-08 19:00:50,808 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, UNASSIGN}, {pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, UNASSIGN}, {pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, UNASSIGN}, {pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, UNASSIGN}, {pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, UNASSIGN}, {pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, UNASSIGN}] 2023-03-08 19:00:50,808 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, UNASSIGN 2023-03-08 19:00:50,809 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, UNASSIGN 2023-03-08 19:00:50,809 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, UNASSIGN 2023-03-08 19:00:50,809 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, UNASSIGN 2023-03-08 19:00:50,809 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, UNASSIGN 2023-03-08 19:00:50,809 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=a8e2ccd674e201b22ff6152ea57faf8e, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:50,809 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302050809"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302050809"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302050809"}]},"ts":"1678302050809"} 2023-03-08 19:00:50,809 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=43584d4ed824c092029e275dd3b82a6c, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:50,809 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=4446f78ec8b0f1645c8eb2d6d276d62d, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:50,809 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302050809"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302050809"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302050809"}]},"ts":"1678302050809"} 2023-03-08 19:00:50,809 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=f98146aa6eba5589f1036e30131dab62, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:50,809 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302050809"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302050809"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302050809"}]},"ts":"1678302050809"} 2023-03-08 19:00:50,809 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302050809"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302050809"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302050809"}]},"ts":"1678302050809"} 2023-03-08 19:00:50,809 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=5b3ed2989f5355f239709c8420b1a06c, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:50,810 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302050809"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302050809"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302050809"}]},"ts":"1678302050809"} 2023-03-08 19:00:50,810 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=277, ppid=276, state=RUNNABLE; CloseRegionProcedure a8e2ccd674e201b22ff6152ea57faf8e, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:50,811 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=278, ppid=273, state=RUNNABLE; CloseRegionProcedure 43584d4ed824c092029e275dd3b82a6c, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:50,811 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=279, ppid=275, state=RUNNABLE; CloseRegionProcedure 4446f78ec8b0f1645c8eb2d6d276d62d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:50,811 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=280, ppid=272, state=RUNNABLE; CloseRegionProcedure f98146aa6eba5589f1036e30131dab62, server=jenkins-hbase6.apache.org,46129,1678302018069}] 2023-03-08 19:00:50,812 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=281, ppid=274, state=RUNNABLE; CloseRegionProcedure 5b3ed2989f5355f239709c8420b1a06c, server=jenkins-hbase6.apache.org,33875,1678302017939}] 2023-03-08 19:00:50,812 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, UNASSIGN 2023-03-08 19:00:50,813 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=f7a0b9e431b1811a0203970d08f9a42d, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:50,813 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302050813"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1678302050813"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1678302050813"}]},"ts":"1678302050813"} 2023-03-08 19:00:50,814 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=282, ppid=271, state=RUNNABLE; CloseRegionProcedure f7a0b9e431b1811a0203970d08f9a42d, server=jenkins-hbase6.apache.org,41161,1678302018011}] 2023-03-08 19:00:50,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:50,962 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing a8e2ccd674e201b22ff6152ea57faf8e, disabling compactions & flushes 2023-03-08 19:00:50,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:50,962 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:50,962 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. after waiting 0 ms 2023-03-08 19:00:50,962 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:50,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing a8e2ccd674e201b22ff6152ea57faf8e 1/1 column families, dataSize=21.92 KB heapSize=47.36 KB 2023-03-08 19:00:50,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:50,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:50,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 5b3ed2989f5355f239709c8420b1a06c, disabling compactions & flushes 2023-03-08 19:00:50,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 4446f78ec8b0f1645c8eb2d6d276d62d, disabling compactions & flushes 2023-03-08 19:00:50,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:50,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:50,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:50,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. after waiting 0 ms 2023-03-08 19:00:50,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:50,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:50,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 5b3ed2989f5355f239709c8420b1a06c 1/1 column families, dataSize=1.63 KB heapSize=3.77 KB 2023-03-08 19:00:50,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. after waiting 0 ms 2023-03-08 19:00:50,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:50,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 4446f78ec8b0f1645c8eb2d6d276d62d 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-03-08 19:00:50,980 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230308292069eb429b4131a05d3fb786df857f to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/e4da3b7fbbce2345d7772b0674a318d520230308292069eb429b4131a05d3fb786df857f 2023-03-08 19:00:50,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202303080581c2f608a0446682f91b941baf122d to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202303080581c2f608a0446682f91b941baf122d 2023-03-08 19:00:50,987 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230308918fdad944fe4db987481f1720ec611f to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/a87ff679a2f3e71d9181a67b7542122c20230308918fdad944fe4db987481f1720ec611f 2023-03-08 19:00:50,990 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=21.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/.tmp/cf/1a971125374f491abd49c3c22f94aaa5 2023-03-08 19:00:50,990 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/.tmp/cf/f96885cc194f4ecca507a42f370d7dcb 2023-03-08 19:00:51,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/.tmp/cf/f96885cc194f4ecca507a42f370d7dcb as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf/f96885cc194f4ecca507a42f370d7dcb 2023-03-08 19:00:51,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/.tmp/cf/1a971125374f491abd49c3c22f94aaa5 as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf/1a971125374f491abd49c3c22f94aaa5 2023-03-08 19:00:51,022 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x594888d9 to 127.0.0.1:59169 2023-03-08 19:00:51,022 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:51,024 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/.tmp/cf/7030864985c94eb2bb01da7b1f2bbda7 2023-03-08 19:00:51,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf/1a971125374f491abd49c3c22f94aaa5, entries=335, sequenceid=15, filesize=62.6 K 2023-03-08 19:00:51,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf/f96885cc194f4ecca507a42f370d7dcb, entries=25, sequenceid=15, filesize=9.2 K 2023-03-08 19:00:51,026 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.63 KB/1673, heapSize ~3.75 KB/3840, currentSize=0 B/0 for 5b3ed2989f5355f239709c8420b1a06c in 63ms, sequenceid=15, compaction requested=false 2023-03-08 19:00:51,028 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~21.92 KB/22441, heapSize ~47.34 KB/48480, currentSize=0 B/0 for a8e2ccd674e201b22ff6152ea57faf8e in 66ms, sequenceid=15, compaction requested=false 2023-03-08 19:00:51,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/.tmp/cf/7030864985c94eb2bb01da7b1f2bbda7 as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf/7030864985c94eb2bb01da7b1f2bbda7 2023-03-08 19:00:51,031 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-03-08 19:00:51,031 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-03-08 19:00:51,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c. 2023-03-08 19:00:51,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 5b3ed2989f5355f239709c8420b1a06c: 2023-03-08 19:00:51,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e. 2023-03-08 19:00:51,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for a8e2ccd674e201b22ff6152ea57faf8e: 2023-03-08 19:00:51,034 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:51,034 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:51,034 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf/7030864985c94eb2bb01da7b1f2bbda7, entries=32, sequenceid=15, filesize=10.4 K 2023-03-08 19:00:51,034 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 43584d4ed824c092029e275dd3b82a6c, disabling compactions & flushes 2023-03-08 19:00:51,034 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:51,034 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:51,034 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. after waiting 0 ms 2023-03-08 19:00:51,034 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:51,034 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=5b3ed2989f5355f239709c8420b1a06c, regionState=CLOSED 2023-03-08 19:00:51,034 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 43584d4ed824c092029e275dd3b82a6c 1/1 column families, dataSize=2.22 KB heapSize=5.03 KB 2023-03-08 19:00:51,034 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302051034"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302051034"}]},"ts":"1678302051034"} 2023-03-08 19:00:51,034 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:51,034 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:51,034 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing f98146aa6eba5589f1036e30131dab62, disabling compactions & flushes 2023-03-08 19:00:51,034 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:51,034 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:51,034 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. after waiting 0 ms 2023-03-08 19:00:51,034 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:51,035 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing f98146aa6eba5589f1036e30131dab62 1/1 column families, dataSize=2.22 KB heapSize=5.03 KB 2023-03-08 19:00:51,035 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 4446f78ec8b0f1645c8eb2d6d276d62d in 72ms, sequenceid=15, compaction requested=false 2023-03-08 19:00:51,035 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=a8e2ccd674e201b22ff6152ea57faf8e, regionState=CLOSED 2023-03-08 19:00:51,035 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302051035"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302051035"}]},"ts":"1678302051035"} 2023-03-08 19:00:51,037 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=281, resume processing ppid=274 2023-03-08 19:00:51,037 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=281, ppid=274, state=SUCCESS; CloseRegionProcedure 5b3ed2989f5355f239709c8420b1a06c, server=jenkins-hbase6.apache.org,33875,1678302017939 in 223 msec 2023-03-08 19:00:51,038 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=274, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=5b3ed2989f5355f239709c8420b1a06c, UNASSIGN in 229 msec 2023-03-08 19:00:51,039 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=277, resume processing ppid=276 2023-03-08 19:00:51,039 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=277, ppid=276, state=SUCCESS; CloseRegionProcedure a8e2ccd674e201b22ff6152ea57faf8e, server=jenkins-hbase6.apache.org,46129,1678302018069 in 226 msec 2023-03-08 19:00:51,039 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=276, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=a8e2ccd674e201b22ff6152ea57faf8e, UNASSIGN in 231 msec 2023-03-08 19:00:51,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-03-08 19:00:51,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d. 2023-03-08 19:00:51,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 4446f78ec8b0f1645c8eb2d6d276d62d: 2023-03-08 19:00:51,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:51,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:51,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing f7a0b9e431b1811a0203970d08f9a42d, disabling compactions & flushes 2023-03-08 19:00:51,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:51,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:51,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. after waiting 0 ms 2023-03-08 19:00:51,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:51,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing f7a0b9e431b1811a0203970d08f9a42d 1/1 column families, dataSize=2.62 KB heapSize=5.88 KB 2023-03-08 19:00:51,048 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=4446f78ec8b0f1645c8eb2d6d276d62d, regionState=CLOSED 2023-03-08 19:00:51,048 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302051048"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302051048"}]},"ts":"1678302051048"} 2023-03-08 19:00:51,050 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=279, resume processing ppid=275 2023-03-08 19:00:51,050 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=279, ppid=275, state=SUCCESS; CloseRegionProcedure 4446f78ec8b0f1645c8eb2d6d276d62d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 238 msec 2023-03-08 19:00:51,051 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=275, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=4446f78ec8b0f1645c8eb2d6d276d62d, UNASSIGN in 242 msec 2023-03-08 19:00:51,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-03-08 19:00:51,076 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230308e103af6bbeb04151a595cb65ac891500 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/c81e728d9d4c2f636f067f89cc14862c20230308e103af6bbeb04151a595cb65ac891500 2023-03-08 19:00:51,079 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230308ba484ff3079444d0a849545c251a9959 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/c4ca4238a0b923820dcc509a6f75849b20230308ba484ff3079444d0a849545c251a9959 2023-03-08 19:00:51,082 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230308d1efc4993d814f44b3dd15e0625e5a32 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/d41d8cd98f00b204e9800998ecf8427e20230308d1efc4993d814f44b3dd15e0625e5a32 2023-03-08 19:00:51,095 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/.tmp/cf/9509f3145138489f958b048d425db591 2023-03-08 19:00:51,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/.tmp/cf/9509f3145138489f958b048d425db591 as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf/9509f3145138489f958b048d425db591 2023-03-08 19:00:51,101 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/.tmp/cf/b32bfebb34124a8993b37b615bbdbcb2 2023-03-08 19:00:51,102 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf/9509f3145138489f958b048d425db591, entries=34, sequenceid=15, filesize=10.8 K 2023-03-08 19:00:51,102 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/.tmp/cf/9ce7efdbae2c4add8235344e069c7f5a 2023-03-08 19:00:51,103 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 43584d4ed824c092029e275dd3b82a6c in 69ms, sequenceid=15, compaction requested=false 2023-03-08 19:00:51,104 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/.tmp/cf/b32bfebb34124a8993b37b615bbdbcb2 as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf/b32bfebb34124a8993b37b615bbdbcb2 2023-03-08 19:00:51,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/.tmp/cf/9ce7efdbae2c4add8235344e069c7f5a as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf/9ce7efdbae2c4add8235344e069c7f5a 2023-03-08 19:00:51,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-03-08 19:00:51,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c. 2023-03-08 19:00:51,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 43584d4ed824c092029e275dd3b82a6c: 2023-03-08 19:00:51,107 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf/b32bfebb34124a8993b37b615bbdbcb2, entries=34, sequenceid=15, filesize=10.8 K 2023-03-08 19:00:51,107 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:51,108 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=43584d4ed824c092029e275dd3b82a6c, regionState=CLOSED 2023-03-08 19:00:51,108 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302051108"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302051108"}]},"ts":"1678302051108"} 2023-03-08 19:00:51,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for f98146aa6eba5589f1036e30131dab62 in 74ms, sequenceid=15, compaction requested=false 2023-03-08 19:00:51,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf/9ce7efdbae2c4add8235344e069c7f5a, entries=40, sequenceid=15, filesize=11.8 K 2023-03-08 19:00:51,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.62 KB/2678, heapSize ~5.86 KB/6000, currentSize=0 B/0 for f7a0b9e431b1811a0203970d08f9a42d in 63ms, sequenceid=15, compaction requested=false 2023-03-08 19:00:51,111 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=278, resume processing ppid=273 2023-03-08 19:00:51,111 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=278, ppid=273, state=SUCCESS; CloseRegionProcedure 43584d4ed824c092029e275dd3b82a6c, server=jenkins-hbase6.apache.org,33875,1678302017939 in 298 msec 2023-03-08 19:00:51,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-03-08 19:00:51,112 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=273, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=43584d4ed824c092029e275dd3b82a6c, UNASSIGN in 303 msec 2023-03-08 19:00:51,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62. 2023-03-08 19:00:51,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for f98146aa6eba5589f1036e30131dab62: 2023-03-08 19:00:51,113 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-03-08 19:00:51,113 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:51,113 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d. 2023-03-08 19:00:51,113 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for f7a0b9e431b1811a0203970d08f9a42d: 2023-03-08 19:00:51,113 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=f98146aa6eba5589f1036e30131dab62, regionState=CLOSED 2023-03-08 19:00:51,113 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1678302051113"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302051113"}]},"ts":"1678302051113"} 2023-03-08 19:00:51,114 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:51,114 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=f7a0b9e431b1811a0203970d08f9a42d, regionState=CLOSED 2023-03-08 19:00:51,115 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1678302051114"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1678302051114"}]},"ts":"1678302051114"} 2023-03-08 19:00:51,115 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=280, resume processing ppid=272 2023-03-08 19:00:51,115 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=280, ppid=272, state=SUCCESS; CloseRegionProcedure f98146aa6eba5589f1036e30131dab62, server=jenkins-hbase6.apache.org,46129,1678302018069 in 303 msec 2023-03-08 19:00:51,116 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=272, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f98146aa6eba5589f1036e30131dab62, UNASSIGN in 307 msec 2023-03-08 19:00:51,116 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=282, resume processing ppid=271 2023-03-08 19:00:51,116 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=282, ppid=271, state=SUCCESS; CloseRegionProcedure f7a0b9e431b1811a0203970d08f9a42d, server=jenkins-hbase6.apache.org,41161,1678302018011 in 301 msec 2023-03-08 19:00:51,117 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=271, resume processing ppid=270 2023-03-08 19:00:51,117 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=271, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1678302045295, region=f7a0b9e431b1811a0203970d08f9a42d, UNASSIGN in 308 msec 2023-03-08 19:00:51,118 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1678302051118"}]},"ts":"1678302051118"} 2023-03-08 19:00:51,119 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1678302045295, state=DISABLED in hbase:meta 2023-03-08 19:00:51,121 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1678302045295 to state=DISABLED 2023-03-08 19:00:51,122 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=270, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1678302045295 in 320 msec 2023-03-08 19:00:51,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-03-08 19:00:51,557 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1678302045295, procId: 270 completed 2023-03-08 19:00:51,558 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.HMaster$5(2228): Client=jenkins//172.31.2.20 delete testCorruptedSnapshot-1678302045295 2023-03-08 19:00:51,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] procedure2.ProcedureExecutor(1029): Stored pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCorruptedSnapshot-1678302045295 2023-03-08 19:00:51,560 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1678302045295 2023-03-08 19:00:51,560 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=283, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1678302045295 2023-03-08 19:00:51,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-03-08 19:00:51,566 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:51,566 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:51,566 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:51,566 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:51,566 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:51,566 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:51,571 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/recovered.edits] 2023-03-08 19:00:51,571 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/recovered.edits] 2023-03-08 19:00:51,571 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/recovered.edits] 2023-03-08 19:00:51,571 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/recovered.edits] 2023-03-08 19:00:51,571 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/recovered.edits] 2023-03-08 19:00:51,572 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf, FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/recovered.edits] 2023-03-08 19:00:51,576 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf/9ce7efdbae2c4add8235344e069c7f5a to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf/9ce7efdbae2c4add8235344e069c7f5a 2023-03-08 19:00:51,576 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf/11aca6c127384ce281223ff929d67805 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf/11aca6c127384ce281223ff929d67805 2023-03-08 19:00:51,576 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf/2b777687c347448d9bce41a3a54b997d to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf/2b777687c347448d9bce41a3a54b997d 2023-03-08 19:00:51,576 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf/2dbc45070fca472eb540554d704c241f to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf/2dbc45070fca472eb540554d704c241f 2023-03-08 19:00:51,576 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf/1a971125374f491abd49c3c22f94aaa5 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf/1a971125374f491abd49c3c22f94aaa5 2023-03-08 19:00:51,577 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf/152f69f176104902b666b5d244c7544a to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf/152f69f176104902b666b5d244c7544a 2023-03-08 19:00:51,578 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf/bb5840f0dc9a4ec99bcb3ce4b4e83dee to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/cf/bb5840f0dc9a4ec99bcb3ce4b4e83dee 2023-03-08 19:00:51,578 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf/b32bfebb34124a8993b37b615bbdbcb2 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/cf/b32bfebb34124a8993b37b615bbdbcb2 2023-03-08 19:00:51,578 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf/9509f3145138489f958b048d425db591 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/cf/9509f3145138489f958b048d425db591 2023-03-08 19:00:51,578 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf/7030864985c94eb2bb01da7b1f2bbda7 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/cf/7030864985c94eb2bb01da7b1f2bbda7 2023-03-08 19:00:51,578 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf/f96885cc194f4ecca507a42f370d7dcb to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/cf/f96885cc194f4ecca507a42f370d7dcb 2023-03-08 19:00:51,578 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf/9922a96e482943fdb71ab70b22b06658 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/cf/9922a96e482943fdb71ab70b22b06658 2023-03-08 19:00:51,581 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/recovered.edits/18.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62/recovered.edits/18.seqid 2023-03-08 19:00:51,581 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/recovered.edits/18.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d/recovered.edits/18.seqid 2023-03-08 19:00:51,582 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/recovered.edits/18.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d/recovered.edits/18.seqid 2023-03-08 19:00:51,582 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/recovered.edits/18.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c/recovered.edits/18.seqid 2023-03-08 19:00:51,582 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f98146aa6eba5589f1036e30131dab62 2023-03-08 19:00:51,582 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/4446f78ec8b0f1645c8eb2d6d276d62d 2023-03-08 19:00:51,582 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/recovered.edits/18.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e/recovered.edits/18.seqid 2023-03-08 19:00:51,582 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/f7a0b9e431b1811a0203970d08f9a42d 2023-03-08 19:00:51,582 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/43584d4ed824c092029e275dd3b82a6c 2023-03-08 19:00:51,583 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/recovered.edits/18.seqid to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c/recovered.edits/18.seqid 2023-03-08 19:00:51,583 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/a8e2ccd674e201b22ff6152ea57faf8e 2023-03-08 19:00:51,583 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/.tmp/data/default/testCorruptedSnapshot-1678302045295/5b3ed2989f5355f239709c8420b1a06c 2023-03-08 19:00:51,583 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1678302045295 regions 2023-03-08 19:00:51,583 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488 2023-03-08 19:00:51,584 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf] 2023-03-08 19:00:51,586 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/a87ff679a2f3e71d9181a67b7542122c20230308918fdad944fe4db987481f1720ec611f to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/a87ff679a2f3e71d9181a67b7542122c20230308918fdad944fe4db987481f1720ec611f 2023-03-08 19:00:51,587 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/c4ca4238a0b923820dcc509a6f75849b202303086b784c2b998442299665458931bcbe8c to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/c4ca4238a0b923820dcc509a6f75849b202303086b784c2b998442299665458931bcbe8c 2023-03-08 19:00:51,588 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/c4ca4238a0b923820dcc509a6f75849b20230308ba484ff3079444d0a849545c251a9959 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/c4ca4238a0b923820dcc509a6f75849b20230308ba484ff3079444d0a849545c251a9959 2023-03-08 19:00:51,589 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/c81e728d9d4c2f636f067f89cc14862c202303085e0db7ce3766480792b5825ff3c6ebb6 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/c81e728d9d4c2f636f067f89cc14862c202303085e0db7ce3766480792b5825ff3c6ebb6 2023-03-08 19:00:51,589 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/c81e728d9d4c2f636f067f89cc14862c20230308e103af6bbeb04151a595cb65ac891500 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/c81e728d9d4c2f636f067f89cc14862c20230308e103af6bbeb04151a595cb65ac891500 2023-03-08 19:00:51,590 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/d41d8cd98f00b204e9800998ecf8427e2023030844f14ff157f64ee1a242d8ff2eb93263 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/d41d8cd98f00b204e9800998ecf8427e2023030844f14ff157f64ee1a242d8ff2eb93263 2023-03-08 19:00:51,591 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/d41d8cd98f00b204e9800998ecf8427e20230308d1efc4993d814f44b3dd15e0625e5a32 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/d41d8cd98f00b204e9800998ecf8427e20230308d1efc4993d814f44b3dd15e0625e5a32 2023-03-08 19:00:51,592 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/e4da3b7fbbce2345d7772b0674a318d520230308224d67a9f09e41b7b978a985bb54751e to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/e4da3b7fbbce2345d7772b0674a318d520230308224d67a9f09e41b7b978a985bb54751e 2023-03-08 19:00:51,593 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/e4da3b7fbbce2345d7772b0674a318d520230308292069eb429b4131a05d3fb786df857f to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/e4da3b7fbbce2345d7772b0674a318d520230308292069eb429b4131a05d3fb786df857f 2023-03-08 19:00:51,594 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202303080581c2f608a0446682f91b941baf122d to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202303080581c2f608a0446682f91b941baf122d 2023-03-08 19:00:51,594 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308e93000cb1d43425da1d8a7cf3446f228 to hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/archive/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230308e93000cb1d43425da1d8a7cf3446f228 2023-03-08 19:00:51,595 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/mobdir/data/default/testCorruptedSnapshot-1678302045295/0a8f983580ce4750649c183a310a7488 2023-03-08 19:00:51,596 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=283, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1678302045295 2023-03-08 19:00:51,598 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testCorruptedSnapshot-1678302045295 from hbase:meta 2023-03-08 19:00:51,600 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testCorruptedSnapshot-1678302045295' descriptor. 2023-03-08 19:00:51,601 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=283, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1678302045295 2023-03-08 19:00:51,601 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testCorruptedSnapshot-1678302045295' from region states. 2023-03-08 19:00:51,601 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302051601"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:51,601 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302051601"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:51,601 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302051601"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:51,601 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302051601"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:51,601 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302051601"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:51,601 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1678302051601"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:51,603 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-03-08 19:00:51,603 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => f7a0b9e431b1811a0203970d08f9a42d, NAME => 'testCorruptedSnapshot-1678302045295,,1678302045297.f7a0b9e431b1811a0203970d08f9a42d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => f98146aa6eba5589f1036e30131dab62, NAME => 'testCorruptedSnapshot-1678302045295,1,1678302045297.f98146aa6eba5589f1036e30131dab62.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 43584d4ed824c092029e275dd3b82a6c, NAME => 'testCorruptedSnapshot-1678302045295,2,1678302045297.43584d4ed824c092029e275dd3b82a6c.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 5b3ed2989f5355f239709c8420b1a06c, NAME => 'testCorruptedSnapshot-1678302045295,3,1678302045297.5b3ed2989f5355f239709c8420b1a06c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 4446f78ec8b0f1645c8eb2d6d276d62d, NAME => 'testCorruptedSnapshot-1678302045295,4,1678302045297.4446f78ec8b0f1645c8eb2d6d276d62d.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => a8e2ccd674e201b22ff6152ea57faf8e, NAME => 'testCorruptedSnapshot-1678302045295,5,1678302045297.a8e2ccd674e201b22ff6152ea57faf8e.', STARTKEY => '5', ENDKEY => ''}] 2023-03-08 19:00:51,603 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testCorruptedSnapshot-1678302045295' as deleted. 2023-03-08 19:00:51,603 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1678302045295","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1678302051603"}]},"ts":"9223372036854775807"} 2023-03-08 19:00:51,604 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testCorruptedSnapshot-1678302045295 state from META 2023-03-08 19:00:51,606 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=283, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1678302045295 2023-03-08 19:00:51,607 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=283, state=SUCCESS; DeleteTableProcedure table=testCorruptedSnapshot-1678302045295 in 48 msec 2023-03-08 19:00:51,741 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-08 19:00:51,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-03-08 19:00:51,812 INFO [Listener at localhost/45021] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testCorruptedSnapshot-1678302045295, procId: 283 completed 2023-03-08 19:00:51,816 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(748): Client=jenkins//172.31.2.20 delete name: "emptySnaptb-1678302045295" 2023-03-08 19:00:51,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1678302045295 2023-03-08 19:00:51,818 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] master.MasterRpcServices(748): Client=jenkins//172.31.2.20 delete name: "snaptb0-1678302045295" 2023-03-08 19:00:51,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43879] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1678302045295 2023-03-08 19:00:51,837 INFO [Listener at localhost/45021] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=421 (was 428), OpenFileDescriptor=807 (was 814), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=305 (was 314), ProcessCount=170 (was 170), AvailableMemoryMB=7450 (was 7486) 2023-03-08 19:00:51,837 INFO [Listener at localhost/45021] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-08 19:00:51,837 INFO [Listener at localhost/45021] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-08 19:00:51,838 DEBUG [Listener at localhost/45021] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x492b920e to 127.0.0.1:59169 2023-03-08 19:00:51,838 DEBUG [Listener at localhost/45021] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:51,838 DEBUG [Listener at localhost/45021] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-08 19:00:51,839 DEBUG [Listener at localhost/45021] util.JVMClusterUtil(257): Found active master hash=1063933363, stopped=false 2023-03-08 19:00:51,839 INFO [Listener at localhost/45021] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase6.apache.org,43879,1678302016597 2023-03-08 19:00:51,843 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:46129-0x101d3eda4f90003, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-08 19:00:51,843 INFO [Listener at localhost/45021] procedure2.ProcedureExecutor(629): Stopping 2023-03-08 19:00:51,843 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-08 19:00:51,843 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-08 19:00:51,843 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-08 19:00:51,843 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:33875-0x101d3eda4f90001, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-08 19:00:51,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46129-0x101d3eda4f90003, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-08 19:00:51,843 DEBUG [Listener at localhost/45021] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x48334728 to 127.0.0.1:59169 2023-03-08 19:00:51,843 DEBUG [Listener at localhost/45021] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:51,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-08 19:00:51,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-08 19:00:51,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33875-0x101d3eda4f90001, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-08 19:00:51,844 INFO [Listener at localhost/45021] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase6.apache.org,33875,1678302017939' ***** 2023-03-08 19:00:51,844 INFO [Listener at localhost/45021] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-08 19:00:51,844 INFO [Listener at localhost/45021] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase6.apache.org,41161,1678302018011' ***** 2023-03-08 19:00:51,844 INFO [Listener at localhost/45021] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-08 19:00:51,844 INFO [Listener at localhost/45021] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase6.apache.org,46129,1678302018069' ***** 2023-03-08 19:00:51,844 INFO [Listener at localhost/45021] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-08 19:00:51,844 INFO [RS:0;jenkins-hbase6:33875] regionserver.HeapMemoryManager(220): Stopping 2023-03-08 19:00:51,844 INFO [RS:1;jenkins-hbase6:41161] regionserver.HeapMemoryManager(220): Stopping 2023-03-08 19:00:51,844 INFO [RS:2;jenkins-hbase6:46129] regionserver.HeapMemoryManager(220): Stopping 2023-03-08 19:00:51,844 INFO [RS:0;jenkins-hbase6:33875] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-08 19:00:51,844 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-08 19:00:51,845 INFO [RS:0;jenkins-hbase6:33875] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-08 19:00:51,844 INFO [RS:2;jenkins-hbase6:46129] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-08 19:00:51,844 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-08 19:00:51,844 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-08 19:00:51,844 INFO [RS:1;jenkins-hbase6:41161] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-08 19:00:51,845 INFO [RS:2;jenkins-hbase6:46129] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-08 19:00:51,845 INFO [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(1145): stopping server jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:51,845 INFO [RS:2;jenkins-hbase6:46129] regionserver.HRegionServer(1145): stopping server jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:51,845 DEBUG [RS:0;jenkins-hbase6:33875] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c108782 to 127.0.0.1:59169 2023-03-08 19:00:51,845 DEBUG [RS:2;jenkins-hbase6:46129] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37d22f28 to 127.0.0.1:59169 2023-03-08 19:00:51,845 DEBUG [RS:0;jenkins-hbase6:33875] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:51,845 DEBUG [RS:2;jenkins-hbase6:46129] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:51,845 INFO [RS:1;jenkins-hbase6:41161] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-08 19:00:51,845 INFO [RS:2;jenkins-hbase6:46129] regionserver.HRegionServer(1171): stopping server jenkins-hbase6.apache.org,46129,1678302018069; all regions closed. 2023-03-08 19:00:51,845 INFO [RS:0;jenkins-hbase6:33875] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-08 19:00:51,845 INFO [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(3304): Received CLOSE for ff6757431e14fac2c9bca54634071a1a 2023-03-08 19:00:51,845 INFO [RS:0;jenkins-hbase6:33875] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-08 19:00:51,845 INFO [RS:0;jenkins-hbase6:33875] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-08 19:00:51,845 INFO [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-08 19:00:51,846 INFO [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(1145): stopping server jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:51,846 DEBUG [RS:1;jenkins-hbase6:41161] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79ce09c8 to 127.0.0.1:59169 2023-03-08 19:00:51,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing ff6757431e14fac2c9bca54634071a1a, disabling compactions & flushes 2023-03-08 19:00:51,846 INFO [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-03-08 19:00:51,846 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a. 2023-03-08 19:00:51,846 DEBUG [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-03-08 19:00:51,846 DEBUG [RS:1;jenkins-hbase6:41161] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:51,846 INFO [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-03-08 19:00:51,846 DEBUG [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(1479): Online Regions={ff6757431e14fac2c9bca54634071a1a=hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a.} 2023-03-08 19:00:51,846 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-08 19:00:51,846 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-08 19:00:51,846 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-08 19:00:51,846 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-08 19:00:51,846 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-08 19:00:51,846 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=137.71 KB heapSize=212.48 KB 2023-03-08 19:00:51,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a. 2023-03-08 19:00:51,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a. after waiting 0 ms 2023-03-08 19:00:51,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a. 2023-03-08 19:00:51,847 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing ff6757431e14fac2c9bca54634071a1a 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-08 19:00:51,848 DEBUG [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(1505): Waiting on ff6757431e14fac2c9bca54634071a1a 2023-03-08 19:00:51,848 DEBUG [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-08 19:00:51,857 DEBUG [RS:2;jenkins-hbase6:46129] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/oldWALs 2023-03-08 19:00:51,857 INFO [RS:2;jenkins-hbase6:46129] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase6.apache.org%2C46129%2C1678302018069:(num 1678302019648) 2023-03-08 19:00:51,857 DEBUG [RS:2;jenkins-hbase6:46129] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:51,857 INFO [RS:2;jenkins-hbase6:46129] regionserver.LeaseManager(133): Closed leases 2023-03-08 19:00:51,857 INFO [RS:2;jenkins-hbase6:46129] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase6:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-03-08 19:00:51,858 INFO [RS:2;jenkins-hbase6:46129] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-08 19:00:51,858 INFO [regionserver/jenkins-hbase6:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-08 19:00:51,858 INFO [RS:2;jenkins-hbase6:46129] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-08 19:00:51,858 INFO [RS:2;jenkins-hbase6:46129] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-08 19:00:51,860 INFO [RS:2;jenkins-hbase6:46129] ipc.NettyRpcServer(158): Stopping server on /172.31.2.20:46129 2023-03-08 19:00:51,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/namespace/ff6757431e14fac2c9bca54634071a1a/.tmp/info/8e5b716f2c9f4d8b8d6c14758bb23f2f 2023-03-08 19:00:51,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/namespace/ff6757431e14fac2c9bca54634071a1a/.tmp/info/8e5b716f2c9f4d8b8d6c14758bb23f2f as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/namespace/ff6757431e14fac2c9bca54634071a1a/info/8e5b716f2c9f4d8b8d6c14758bb23f2f 2023-03-08 19:00:51,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/namespace/ff6757431e14fac2c9bca54634071a1a/info/8e5b716f2c9f4d8b8d6c14758bb23f2f, entries=2, sequenceid=6, filesize=4.8 K 2023-03-08 19:00:51,870 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for ff6757431e14fac2c9bca54634071a1a in 24ms, sequenceid=6, compaction requested=false 2023-03-08 19:00:51,874 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=130.94 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/.tmp/info/3aa5d1f533aa48f5b16359f85fbb23a9 2023-03-08 19:00:51,877 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/namespace/ff6757431e14fac2c9bca54634071a1a/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-08 19:00:51,879 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3aa5d1f533aa48f5b16359f85fbb23a9 2023-03-08 19:00:51,879 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a. 2023-03-08 19:00:51,879 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for ff6757431e14fac2c9bca54634071a1a: 2023-03-08 19:00:51,879 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1678302020039.ff6757431e14fac2c9bca54634071a1a. 2023-03-08 19:00:51,888 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.01 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/.tmp/rep_barrier/4ae028c34ee24c2e9b329ae3485cd908 2023-03-08 19:00:51,890 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4ae028c34ee24c2e9b329ae3485cd908 2023-03-08 19:00:51,902 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.75 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/.tmp/table/028131a734e0498c98a7c1625f2715be 2023-03-08 19:00:51,905 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 028131a734e0498c98a7c1625f2715be 2023-03-08 19:00:51,905 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/.tmp/info/3aa5d1f533aa48f5b16359f85fbb23a9 as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/info/3aa5d1f533aa48f5b16359f85fbb23a9 2023-03-08 19:00:51,908 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3aa5d1f533aa48f5b16359f85fbb23a9 2023-03-08 19:00:51,908 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/info/3aa5d1f533aa48f5b16359f85fbb23a9, entries=64, sequenceid=307, filesize=11.8 K 2023-03-08 19:00:51,909 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/.tmp/rep_barrier/4ae028c34ee24c2e9b329ae3485cd908 as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/rep_barrier/4ae028c34ee24c2e9b329ae3485cd908 2023-03-08 19:00:51,912 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4ae028c34ee24c2e9b329ae3485cd908 2023-03-08 19:00:51,912 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/rep_barrier/4ae028c34ee24c2e9b329ae3485cd908, entries=18, sequenceid=307, filesize=7.0 K 2023-03-08 19:00:51,913 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/.tmp/table/028131a734e0498c98a7c1625f2715be as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/table/028131a734e0498c98a7c1625f2715be 2023-03-08 19:00:51,915 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 028131a734e0498c98a7c1625f2715be 2023-03-08 19:00:51,915 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/table/028131a734e0498c98a7c1625f2715be, entries=22, sequenceid=307, filesize=7.0 K 2023-03-08 19:00:51,916 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 70ms, sequenceid=307, compaction requested=false 2023-03-08 19:00:51,920 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/data/hbase/meta/1588230740/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=1 2023-03-08 19:00:51,921 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-08 19:00:51,921 INFO [regionserver/jenkins-hbase6:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-08 19:00:51,921 INFO [regionserver/jenkins-hbase6:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-08 19:00:51,921 INFO [regionserver/jenkins-hbase6:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-08 19:00:51,922 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-08 19:00:51,922 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-08 19:00:51,922 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-08 19:00:51,952 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:51,952 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:46129-0x101d3eda4f90003, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:51,952 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-08 19:00:51,952 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:33875-0x101d3eda4f90001, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase6.apache.org,46129,1678302018069 2023-03-08 19:00:51,952 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-08 19:00:51,953 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:33875-0x101d3eda4f90001, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-08 19:00:51,953 ERROR [Listener at localhost/45021-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2e8789f2 rejected from java.util.concurrent.ThreadPoolExecutor@2ab818fa[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-08 19:00:51,953 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:46129-0x101d3eda4f90003, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-08 19:00:51,954 ERROR [Listener at localhost/45021-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4c456197 rejected from java.util.concurrent.ThreadPoolExecutor@2ab818fa[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-08 19:00:51,954 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase6.apache.org,46129,1678302018069] 2023-03-08 19:00:51,954 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase6.apache.org,46129,1678302018069; numProcessing=1 2023-03-08 19:00:51,956 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase6.apache.org,46129,1678302018069 already deleted, retry=false 2023-03-08 19:00:51,956 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase6.apache.org,46129,1678302018069 expired; onlineServers=2 2023-03-08 19:00:52,048 INFO [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(1171): stopping server jenkins-hbase6.apache.org,41161,1678302018011; all regions closed. 2023-03-08 19:00:52,050 INFO [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(1171): stopping server jenkins-hbase6.apache.org,33875,1678302017939; all regions closed. 2023-03-08 19:00:52,054 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/WALs/jenkins-hbase6.apache.org,41161,1678302018011/jenkins-hbase6.apache.org%2C41161%2C1678302018011.1678302019647 not finished, retry = 0 2023-03-08 19:00:52,056 DEBUG [RS:0;jenkins-hbase6:33875] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/oldWALs 2023-03-08 19:00:52,057 INFO [RS:0;jenkins-hbase6:33875] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase6.apache.org%2C33875%2C1678302017939.meta:.meta(num 1678302019776) 2023-03-08 19:00:52,061 DEBUG [RS:0;jenkins-hbase6:33875] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/oldWALs 2023-03-08 19:00:52,061 INFO [RS:0;jenkins-hbase6:33875] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase6.apache.org%2C33875%2C1678302017939:(num 1678302019648) 2023-03-08 19:00:52,061 DEBUG [RS:0;jenkins-hbase6:33875] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:52,061 INFO [RS:0;jenkins-hbase6:33875] regionserver.LeaseManager(133): Closed leases 2023-03-08 19:00:52,062 INFO [RS:0;jenkins-hbase6:33875] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase6:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-03-08 19:00:52,062 INFO [regionserver/jenkins-hbase6:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-08 19:00:52,063 INFO [RS:0;jenkins-hbase6:33875] ipc.NettyRpcServer(158): Stopping server on /172.31.2.20:33875 2023-03-08 19:00:52,065 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:33875-0x101d3eda4f90001, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:52,065 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-08 19:00:52,065 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase6.apache.org,33875,1678302017939 2023-03-08 19:00:52,065 ERROR [Listener at localhost/45021-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1258ccec rejected from java.util.concurrent.ThreadPoolExecutor@16675a7f[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-08 19:00:52,066 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase6.apache.org,33875,1678302017939] 2023-03-08 19:00:52,066 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase6.apache.org,33875,1678302017939; numProcessing=2 2023-03-08 19:00:52,067 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase6.apache.org,33875,1678302017939 already deleted, retry=false 2023-03-08 19:00:52,067 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase6.apache.org,33875,1678302017939 expired; onlineServers=1 2023-03-08 19:00:52,157 DEBUG [RS:1;jenkins-hbase6:41161] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/oldWALs 2023-03-08 19:00:52,157 INFO [RS:1;jenkins-hbase6:41161] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase6.apache.org%2C41161%2C1678302018011:(num 1678302019647) 2023-03-08 19:00:52,157 DEBUG [RS:1;jenkins-hbase6:41161] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:52,157 INFO [RS:1;jenkins-hbase6:41161] regionserver.LeaseManager(133): Closed leases 2023-03-08 19:00:52,157 INFO [RS:1;jenkins-hbase6:41161] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase6:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-03-08 19:00:52,157 INFO [RS:1;jenkins-hbase6:41161] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-08 19:00:52,157 INFO [RS:1;jenkins-hbase6:41161] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-08 19:00:52,157 INFO [RS:1;jenkins-hbase6:41161] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-08 19:00:52,157 INFO [regionserver/jenkins-hbase6:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-08 19:00:52,158 INFO [RS:1;jenkins-hbase6:41161] ipc.NettyRpcServer(158): Stopping server on /172.31.2.20:41161 2023-03-08 19:00:52,161 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-08 19:00:52,161 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase6.apache.org,41161,1678302018011 2023-03-08 19:00:52,161 ERROR [Listener at localhost/45021-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5c1c41a7 rejected from java.util.concurrent.ThreadPoolExecutor@5823cefc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-08 19:00:52,162 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase6.apache.org,41161,1678302018011] 2023-03-08 19:00:52,162 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase6.apache.org,41161,1678302018011; numProcessing=3 2023-03-08 19:00:52,163 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase6.apache.org,41161,1678302018011 already deleted, retry=false 2023-03-08 19:00:52,163 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase6.apache.org,41161,1678302018011 expired; onlineServers=0 2023-03-08 19:00:52,163 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase6.apache.org,43879,1678302016597' ***** 2023-03-08 19:00:52,163 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-08 19:00:52,164 DEBUG [M:0;jenkins-hbase6:43879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e575f88, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase6.apache.org/172.31.2.20:0 2023-03-08 19:00:52,164 INFO [M:0;jenkins-hbase6:43879] regionserver.HRegionServer(1145): stopping server jenkins-hbase6.apache.org,43879,1678302016597 2023-03-08 19:00:52,164 INFO [M:0;jenkins-hbase6:43879] regionserver.HRegionServer(1171): stopping server jenkins-hbase6.apache.org,43879,1678302016597; all regions closed. 2023-03-08 19:00:52,164 DEBUG [M:0;jenkins-hbase6:43879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-08 19:00:52,164 DEBUG [M:0;jenkins-hbase6:43879] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-08 19:00:52,164 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-08 19:00:52,165 DEBUG [M:0;jenkins-hbase6:43879] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-08 19:00:52,165 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster-HFileCleaner.small.0-1678302019274] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase6:0:becomeActiveMaster-HFileCleaner.small.0-1678302019274,5,FailOnTimeoutGroup] 2023-03-08 19:00:52,165 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster-HFileCleaner.large.0-1678302019273] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase6:0:becomeActiveMaster-HFileCleaner.large.0-1678302019273,5,FailOnTimeoutGroup] 2023-03-08 19:00:52,165 INFO [M:0;jenkins-hbase6:43879] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-08 19:00:52,165 INFO [M:0;jenkins-hbase6:43879] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-08 19:00:52,165 INFO [M:0;jenkins-hbase6:43879] hbase.ChoreService(369): Chore service for: master/jenkins-hbase6:0 had [] on shutdown 2023-03-08 19:00:52,165 DEBUG [M:0;jenkins-hbase6:43879] master.HMaster(1512): Stopping service threads 2023-03-08 19:00:52,165 INFO [M:0;jenkins-hbase6:43879] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-08 19:00:52,166 ERROR [M:0;jenkins-hbase6:43879] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] 2023-03-08 19:00:52,166 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-03-08 19:00:52,166 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-08 19:00:52,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-08 19:00:52,166 INFO [M:0;jenkins-hbase6:43879] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-08 19:00:52,166 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-08 19:00:52,167 DEBUG [M:0;jenkins-hbase6:43879] zookeeper.ZKUtil(398): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-08 19:00:52,167 WARN [M:0;jenkins-hbase6:43879] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-08 19:00:52,167 INFO [M:0;jenkins-hbase6:43879] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-08 19:00:52,167 INFO [M:0;jenkins-hbase6:43879] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-08 19:00:52,167 DEBUG [M:0;jenkins-hbase6:43879] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-08 19:00:52,167 INFO [M:0;jenkins-hbase6:43879] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-08 19:00:52,167 DEBUG [M:0;jenkins-hbase6:43879] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-08 19:00:52,167 DEBUG [M:0;jenkins-hbase6:43879] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-08 19:00:52,167 DEBUG [M:0;jenkins-hbase6:43879] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-08 19:00:52,167 INFO [M:0;jenkins-hbase6:43879] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=845.46 KB heapSize=1.01 MB 2023-03-08 19:00:52,179 INFO [M:0;jenkins-hbase6:43879] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=845.46 KB at sequenceid=2109 (bloomFilter=true), to=hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/be171d00bcc94e8ea31a3125862349dd 2023-03-08 19:00:52,182 INFO [M:0;jenkins-hbase6:43879] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for be171d00bcc94e8ea31a3125862349dd 2023-03-08 19:00:52,183 DEBUG [M:0;jenkins-hbase6:43879] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/be171d00bcc94e8ea31a3125862349dd as hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/be171d00bcc94e8ea31a3125862349dd 2023-03-08 19:00:52,185 INFO [M:0;jenkins-hbase6:43879] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for be171d00bcc94e8ea31a3125862349dd 2023-03-08 19:00:52,185 INFO [M:0;jenkins-hbase6:43879] regionserver.HStore(1080): Added hdfs://localhost:34283/user/jenkins/test-data/470026b9-bdfa-13ba-303f-7191dd9528b1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/be171d00bcc94e8ea31a3125862349dd, entries=282, sequenceid=2109, filesize=30.9 K 2023-03-08 19:00:52,186 INFO [M:0;jenkins-hbase6:43879] regionserver.HRegion(2948): Finished flush of dataSize ~845.46 KB/865749, heapSize ~1.01 MB/1061760, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 19ms, sequenceid=2109, compaction requested=false 2023-03-08 19:00:52,187 INFO [M:0;jenkins-hbase6:43879] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-08 19:00:52,187 DEBUG [M:0;jenkins-hbase6:43879] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-08 19:00:52,190 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-08 19:00:52,190 INFO [M:0;jenkins-hbase6:43879] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-08 19:00:52,191 INFO [M:0;jenkins-hbase6:43879] ipc.NettyRpcServer(158): Stopping server on /172.31.2.20:43879 2023-03-08 19:00:52,194 DEBUG [M:0;jenkins-hbase6:43879] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase6.apache.org,43879,1678302016597 already deleted, retry=false 2023-03-08 19:00:52,343 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-08 19:00:52,343 INFO [M:0;jenkins-hbase6:43879] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase6.apache.org,43879,1678302016597; zookeeper connection closed. 2023-03-08 19:00:52,343 ERROR [Listener at localhost/45021-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3c55759f rejected from java.util.concurrent.ThreadPoolExecutor@70da0bc9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-08 19:00:52,343 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): master:43879-0x101d3eda4f90000, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-08 19:00:52,343 ERROR [Listener at localhost/45021-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5c94e218 rejected from java.util.concurrent.ThreadPoolExecutor@70da0bc9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-08 19:00:52,443 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-08 19:00:52,443 INFO [RS:1;jenkins-hbase6:41161] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase6.apache.org,41161,1678302018011; zookeeper connection closed. 2023-03-08 19:00:52,443 ERROR [Listener at localhost/45021-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@12d97a71 rejected from java.util.concurrent.ThreadPoolExecutor@5823cefc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-08 19:00:52,443 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:41161-0x101d3eda4f90002, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-08 19:00:52,443 ERROR [Listener at localhost/45021-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6a647fa8 rejected from java.util.concurrent.ThreadPoolExecutor@5823cefc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-08 19:00:52,443 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2ed5185a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2ed5185a 2023-03-08 19:00:52,543 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:33875-0x101d3eda4f90001, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-08 19:00:52,543 INFO [RS:0;jenkins-hbase6:33875] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase6.apache.org,33875,1678302017939; zookeeper connection closed. 2023-03-08 19:00:52,543 ERROR [Listener at localhost/45021-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@369782fd rejected from java.util.concurrent.ThreadPoolExecutor@16675a7f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-08 19:00:52,543 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:33875-0x101d3eda4f90001, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-08 19:00:52,543 ERROR [Listener at localhost/45021-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4ee40ded rejected from java.util.concurrent.ThreadPoolExecutor@16675a7f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-08 19:00:52,543 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@34c7020f] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@34c7020f 2023-03-08 19:00:52,643 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:46129-0x101d3eda4f90003, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-08 19:00:52,643 INFO [RS:2;jenkins-hbase6:46129] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase6.apache.org,46129,1678302018069; zookeeper connection closed. 2023-03-08 19:00:52,643 ERROR [Listener at localhost/45021-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@340a1505 rejected from java.util.concurrent.ThreadPoolExecutor@2ab818fa[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-08 19:00:52,644 DEBUG [Listener at localhost/45021-EventThread] zookeeper.ZKWatcher(600): regionserver:46129-0x101d3eda4f90003, quorum=127.0.0.1:59169, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-08 19:00:52,644 ERROR [Listener at localhost/45021-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@683bed3a rejected from java.util.concurrent.ThreadPoolExecutor@2ab818fa[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-08 19:00:52,644 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1470bbfc] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1470bbfc 2023-03-08 19:00:52,644 INFO [Listener at localhost/45021] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-03-08 19:00:52,644 WARN [Listener at localhost/45021] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-08 19:00:52,648 INFO [Listener at localhost/45021] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-08 19:00:52,753 WARN [BP-1648455922-172.31.2.20-1678302013103 heartbeating to localhost/127.0.0.1:34283] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-08 19:00:52,753 WARN [BP-1648455922-172.31.2.20-1678302013103 heartbeating to localhost/127.0.0.1:34283] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1648455922-172.31.2.20-1678302013103 (Datanode Uuid 7368293d-a0de-467d-aff9-0b753c131e12) service to localhost/127.0.0.1:34283 2023-03-08 19:00:52,755 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/cluster_ea2ca4d9-6781-beee-9c1a-51bfaa940573/dfs/data/data5/current/BP-1648455922-172.31.2.20-1678302013103] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-08 19:00:52,755 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/cluster_ea2ca4d9-6781-beee-9c1a-51bfaa940573/dfs/data/data6/current/BP-1648455922-172.31.2.20-1678302013103] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-08 19:00:52,756 WARN [Listener at localhost/45021] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-08 19:00:52,758 INFO [Listener at localhost/45021] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-08 19:00:52,863 WARN [BP-1648455922-172.31.2.20-1678302013103 heartbeating to localhost/127.0.0.1:34283] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-08 19:00:52,863 WARN [BP-1648455922-172.31.2.20-1678302013103 heartbeating to localhost/127.0.0.1:34283] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1648455922-172.31.2.20-1678302013103 (Datanode Uuid 24830eb0-72be-48b6-9e8c-ace0fc6eb977) service to localhost/127.0.0.1:34283 2023-03-08 19:00:52,863 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/cluster_ea2ca4d9-6781-beee-9c1a-51bfaa940573/dfs/data/data3/current/BP-1648455922-172.31.2.20-1678302013103] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-08 19:00:52,864 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/cluster_ea2ca4d9-6781-beee-9c1a-51bfaa940573/dfs/data/data4/current/BP-1648455922-172.31.2.20-1678302013103] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-08 19:00:52,864 WARN [Listener at localhost/45021] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-08 19:00:52,866 INFO [Listener at localhost/45021] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-08 19:00:52,971 WARN [BP-1648455922-172.31.2.20-1678302013103 heartbeating to localhost/127.0.0.1:34283] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-08 19:00:52,971 WARN [BP-1648455922-172.31.2.20-1678302013103 heartbeating to localhost/127.0.0.1:34283] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1648455922-172.31.2.20-1678302013103 (Datanode Uuid 9513f804-42c9-41da-863a-7ca390f59556) service to localhost/127.0.0.1:34283 2023-03-08 19:00:52,971 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/cluster_ea2ca4d9-6781-beee-9c1a-51bfaa940573/dfs/data/data1/current/BP-1648455922-172.31.2.20-1678302013103] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-08 19:00:52,972 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b7272ab4-29c0-ea76-0d18-a07c123411a4/cluster_ea2ca4d9-6781-beee-9c1a-51bfaa940573/dfs/data/data2/current/BP-1648455922-172.31.2.20-1678302013103] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-08 19:00:52,999 INFO [Listener at localhost/45021] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-08 19:00:53,131 INFO [Listener at localhost/45021] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-08 19:00:53,226 INFO [Listener at localhost/45021] hbase.HBaseTestingUtility(1293): Minicluster is down